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Addressing the challenge of inequality has been the ‘holy grail’ of global 
development programmes, and scholars and practitioners across various 
fields, such as economics, development, sociology and politics, have 
studied and ruminated on the solution to the pernicious challenge for 
decades. Almost 20 years later, the persistent resonance of the title of the 
book, When will we ever learn, by the Centre for Global Development 
(2006), is a stark reminder that though there has been a rapid increase in 
the availability of data on inequality, the evidence of ‘what works’, ‘what 
does not work’, ‘for whom’ and ‘under what conditions’ is much more 
complex. Despite the volumes of evidence that provide seemingly 
straightforward answers around, for example, the impact of cash transfer 
programmes, agricultural development initiatives or measures to improve 
access to health and education to underserved groups, the world continues 
to battle the protracted challenge of global inequalities. The World 
Inequality Report (2022)! states that, in terms of income inequality: 

[...] in every large region of the world with the exception of Europe, the share of 

the bottom 50% in total earnings is less than 15% (less than ten in Latin America, 


Sub-Saharan Africa and MENA region) while the share of the richest 10% is over 
40% and in many of the regions, closer to 60%. (n.p.) 


Wealth inequality reveals similarly startling figures. The impact of the 
climate crisis has exacerbated the challenge, with the World Inequality 
Database (2023)? conveying the unequal distribution of climate impacts 
across the world and low- and middle-income countries suffering much 
more than their wealthier counterparts. This book, therefore, remains a 
necessary contribution to the continued advocacy and push needed to 
address the last frontier of colonialism and underdevelopment, and the 
need for global action across all sectors, by all actors, to dismantle the 


1. See https://wir2022.wid.world/download/. 


2. See https://wid.world/news-article/climate-inequality-report-2023-fair-taxes-for-a-sustainable-future-in-the- 
global-south/. 
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harmful systems, practices and institutions that create the oppressive and 
protracted conditions for inequality to persist. 


While the subject of this book is not new, the focus of the collection - 
the link between evaluation and inequality - to a certain extent is, and has 
been, left trailing behind research and reflections on inequality in 
development. Several authors from the Global South have therefore come 
together to examine the relationship between equity (fair distribution of 
wealth and resources), systemic inequality (the institutionalised, unfair 
treatment of people based on attributes such as race, gender, sexual 
orientation, religion, class and social status) and evaluation. It forms part of 
the growing discourse around transforming evaluation (which has also 
received a more candid focus on decolonising and indigenising evaluation 
as key factors that are fundamental to the need to transform in recent 
years). Over the last decade or so, evaluators have slowly begun addressing 
the issue of transforming evaluation and considering more than simply the 
success or failure of individual programmes on social change. This book 
presents a contribution to this and is a balanced collection that addresses 
the systemic nature of inequity and inequality, as well as the microcosm of 
evaluation practice and its role in supporting equitable development. For 
example, Chapters 1 and 3 unpack the global hegemony of the so-called 
Global North vis-a-vis the Global South and the historical power asymmetries 
that have their roots in colonialism and underdevelopment as introductions 
to understanding inequality and its manifestations. Chapter 4 challenges 
us to look more closely at the institutions that are the ‘guardians’ of 
development and the need to address unchecked hegemonic power in 
some cases. Chapter 2 considers what an authentic practice and living the 
principles of ‘Made in Africa Evaluation’ (MAE) would look like, and 
Chapter 7, for example, examines what works in addressing the issue of 
gender-based violence by learning from a South African evaluation of 
interventions. 


The key question that this book raises for all of us is: how can our practice 
and study of evaluation shift the needle and achieve equity and equality for 
all? Our hope is in the persevering work of champions in government, 
parliaments, the private sector, civil society, international development 
organisations, academic institutions, research institutes, think tanks and 
numerous community-based organisations to do the work that is needed 
to tear down the oppressive systems and practices that generate inequality 
and to scale up the actions that are needed to achieve the goal of inclusive 
growth and development. For evaluators, the labour is to produce and 
work tirelessly towards the effective use of the evidence needed to do so. 
This book, therefore, stands as a contribution to the active and persistent 
work of the pursuit of social justice and the role of evaluation as a catalyst. 
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The World Health Organization (WHO) (2023, n.p.) defines equity as ‘the 
absence of avoidable or remediable differences among groups of people, 
whether those groups are defined socially, economically, demographically 
or geographically’. The goal of equity is to eliminate the unfair and avoidable 
circumstances that deprive people of their rights. Therefore, inequities 
generally arise when certain population groups are unfairly deprived of 
basic resources that are made available to other groups. A disparity is 
‘unfair’ or ‘unjust’ when it is the result of the social context rather than 
biological factors. Equitable evaluation contends that conducting evaluation 
practices with an equity approach is more powerful, as evaluation is used 
as a tool for advancing equity. It emphasises that context, culture, history 
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and beliefs shape the nature of evaluations, specifically in the diverse and 
often complex African reality. Furthermore, equitable evaluation can render 
power to the powerless, offer a voice to the silenced and give presence to 
those treated as invisible. 


Evidence from various sources shows that inequality is prevalent in the 
African continent, hence the need to focus on evaluative solutions that 
address the structural issues that contribute to the different forms of 
inequality, such as economic, political and social inequality. Despite a 
plethora of development interventions in the African continent, a large 
proportion of the population on the continent still lacks access to basic 
goods and services for survival. The effectiveness of developmental 
programmes in sub-Saharan Africa has been elusive, to the extent that 
minimal inroads have been made in addressing key challenges such as 
poverty, inequality and, currently, the effects of climate change. The Centre 
for Learning on Evaluation and Results in Anglophone Africa (CLEAR-AA), 
supported by Ford Foundation, commissioned two volumes on equitable 
evaluation in the Global South. 


The two volumes seek to stimulate conversations among development 
evaluators, commissioners of evaluations and development programme 
decision-makers about the role of evaluation in addressing inequality and 
fostering an equitable society in Africa. The book chapters explore the 
following: (1) Take stock of what we know about inequality - what is 
inequality in the African context, and how does it affect the lives of the 
citizens of African countries? (2) What does equitable evaluation mean? In 
what way can the concept of equitable evaluation be adopted in evaluation 
practice? (3) What lessons can be learnt from evaluations of interventions 
that address inequality at various (sectoral, programmatic and project) 
levels? (4) What epistemological transformation in evaluation practice is 
needed to achieve an equitable society? (5) How have issues of inequality 
manifested within evaluation practice through organisations, institutions 
and international development? 


In Chapter 1, Masvaure, Fish, Chirau, Morkel and Mkhize introduce the 
key terms that are key to understanding inequality. The chapter critically 
discusses inequality, inequity, equity, equality and the meaning of equitable 
evaluation. The chapter links inequality to the current development 
challenges in sub-Saharan Africa. Furthermore, the chapter examines the 
root causes of inequality in the Global South through situating inequality in 
the colonial history of the Global South. It examines how colonial history 
shaped the current levels of inequality in the Global South. The chapter 
concludes by highlighting the role evaluation plays in addressing inequality. 


In Chapter 2, Murgatroyd and Feront explore how a Made in Africa 
Evaluation (MAE) approach can enhance positive impacts for all stakeholders. 


Introduction 


They argue that evaluations in Africa are typically conducted to serve the 
needs and interests of donors rather than local beneficiaries. This results in 
an extractive process whereby programme participants become the objects 
of data instead of being empowered to be active contributors in data- 
collection and analysis processes. The chapter is an application of the MAE 
approach and explores the experiences of key stakeholders in two youth 
development programmes operated by the African Foundation in rural 
KwaZulu-Natal, South Africa. The key findings from the chapter are that 
participants’ individual expectations of the development programme shape 
their experience of success or failure; that evaluations need to consider the 
impact on not only the individual but also their family, community and the 
environment; and that there is a willingness among participants to select 
the best elements of both a Western and Zulu culture to create an evaluation 
framework that best reflects their lived experiences. Further, they found 
that there is a growing number of donors who, if guided and supported 
through the process, are willing to adapt their current evaluation frameworks 
to better suit the African context. 


In Chapter 3, Dlakavu argues that evaluation theory (transmitted through 
African institutions) and practice are dominated by Western neoliberal 
conceptualisations of what constitutes ‘development’ and how to measure 
it. Such Western monopolisation of evaluation knowledge and practice 
is a reinforcement of the Global North’s neo-colonial dominance in 
Africa ideologically (inclusive of ideas of development), politically and 
economically. This chapter examines the dominance of Global North 
development theory and practice within African university curricula, African 
governments and non-state developmental stakeholders. He concludes 
this chapter by providing recommendations on how to dismantle the Global 
North’s monopoly of development and monitoring and evaluation (M&E) 
discourse and practice. 


In Chapter 4, Chirau and Umali argue that Marxist analysis of class offers 
a better understanding of how inequality emerged and sustained itself 
even to contemporary times. The class struggle is replicated by international 
institutions, albeit in new forms. Institutions such as the United Nations 
(UN) and its agents, the World Bank and the International Monetary Fund 
(IMF) are no exception in perpetuating inequalities, particularly in the 
Global South. This is seen in how the Global North relates to the Global 
South in multilateral organisations. Chirau and Umali expose the hidden 
sources of inequality manifesting in the evaluation practice and space. The 
chapter hypothesises that the main nucleus of inequity and inequalities 
existing in M&E practices are traced to the operations as well as attitudes 
of the international institutions towards the Global South because these 
provide advice and recommend interventions. Their chapter concludes 
that there is no doubt that international organisations play an important 
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role in eliminating inequities and inequalities, but they must be aware of 
their impact to prevent unintentionally reinforcing inequalities and 
inequities. The chapter ends by providing recommendations that include 
the role of MAE and indigenous knowledge systems (IKSs) in reducing the 
epistemological inequality between the Global North and Global South. 


In Chapter 5, Masvaure demonstrates how the use of the intersectional 
approach in evaluation contributes to a better understanding of the climate 
injustice problem. Furthermore, the chapter explores how the various forms 
of inequalities influence climate change adaptation decision-making. The 
chapter adopts autoethnography and tells a compelling story of how one 
individual is affected by different types of inequities. The chapter deals in 
detail with the systematic nature of inequality and how individuals, 
communities and countries are affected by multi-dimensions of inequality. 
Masvaure emphasised the need for a holistic approach to addressing 
inequities in development if sustainable and impactful development is to 
be achieved. The chapter concludes by calling evaluators to action by 
employing appropriate approaches that address these multi-dimensions of 
inequalities. 


In Chapter 6, Mkhize argues that evaluators’ interrogation of the 
persistent colonial and neo-colonial agenda inherent in the framing of 
climate-smart agriculture programming. Her chapter reveals how CSA 
programming pursues distributive justice with limited forms of procedural, 
restorative and intergenerational justice, which remains largely unquestioned 
in evaluation. She illustrates how critical interrogation of confining framings 
and confronting injustices of climate change narratives and ideologies in 
development and evaluation can provide evidence for rethinking how CSA 
programming and evaluation are conducted in Africa. Her chapter also 
provides limitations and opportunities for evaluators seeking to centre 
equity in evaluative practice. The chapter highlights various evaluation 
approaches and methodologies that evaluators can employ to challenge 
politics, power and ideology. It concludes by cautioning against the view 
that evaluation alone can bring equity and justice in the highly political and 
contested CSA discourses in Africa. 


In Chapter 7, Amisi and Parenzee explore the applicability and 
usefulness of gender inequality as the focus of interventions to prevent 
GBV in South Africa with its history of colonial and racial subjugation. The 
chapter synthesised a sample of intervention programme evaluations in a 
2019 Evidence Map carried out by CLEAR-AA, the Institute for Security 
Studies and the Africa Centre of Evidence using the ‘What is the problem 
represented to be (WPR)’ analytical framework. The chapter highlights 
that evaluators focus on programmes as the unit of analysis in an 
evaluation, and this leads to a narrow definition of gender inequality. 
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Furthermore, evaluators are often guided by what the programme 
identified as a problem and what they intended to do to address the 
problem. They concluded by saying that it is imperative that we open up 
debates about how we understand gender inequality and its relationship 
to the forms of violence that manifest in public and private spaces. These 
debates present an opportunity to forge new analytical perspectives that 
reflect the lived experiences of those who occupy a given context. 


In Chapter 8, Fish argues that evaluation is needed to facilitate the 
prioritisation of mental health equity in African countries. Fish stresses that 
mental health care is low on the public health agenda, as demonstrated by 
the scarcity of mental health care specialists, the existence of only a few 
outpatient and inpatient health care facilities or psychiatric wards in general 
hospitals with sufficient beds for mentally ill patients. One of the primary 
interventions in addressing this treatment gap in African countries is task- 
sharing (i.e. where non-specialist health professionals and lay workers 
receive training and supervision to screen for or diagnose mental illness 
and to provide treatment and monitor affected people). This chapter shows 
that the evaluations conducted in Africa available on the African Evaluation 
Database (AfrED) suggest that there are two different ways in which task- 
sharing interventions have been evaluated, using RCTs and a theory-based 
evaluation approach, both of which were problematic. The exclusionary 
criteria used for the RCTs meant that the success or failure of the 
interventions was measured using data from populations which were easier 
to study, and the effect on the most marginalised people, including the 
elderly and children from rural areas and those with severe mental illness, 
was ignored. The theory-based approach, on the other hand, did not use 
change theories from psychiatric or psychological fields to support the 
explanation of how change occurs. She concludes the chapter by stating 
that equity-focused evaluations should go beyond showing results only 
from the medical perspective but need to also consider the influence 
of social contexts on the problems themselves and the effectiveness of 
treatment. 


In Chapter 9, Grand and Mutereko demonstrate how critical discourse 
analysis presents theoretical and methodological opportunities to identify 
inequalities in evaluations and their implications for policy and practice in 
Zimbabwe. The chapter indicates both negative and unintended impacts, 
including discourses promoting the exclusion of local voices from key 
evaluation findings. Excessive reliance on global health partnerships’ financial 
and technical support has excluded local M&E knowledge systems as 
quantitative techniques, policies, indicators and targets define knowledge. 
The M&E partnership discourse also normalises the unequal power relations 
in which the government is presumed the weaker partner - an arrangement 
exploited, to their own advantage, by both parties. Consequently, the chapter 
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argues that there is a need for deliberate scientific and pragmatic discussions 
that problematise evaluation approaches supported through global health 
partnerships, such as results- and evidence-based evaluations explicitly 
focusing on its unintended effects. 


In Chapter 10, Muller tackles the well-known concerns of randomised 
control trials (RCTs). He argues that the manner in which RCTs are being 
used could hinder social and economic development rather than advance 
it. The chapter focuses, in particular, on three RCT concerns. He argues that 
while RCTs are presented as ‘scientific’ and essentially value-free, they are 
heavily influenced by the worldview of those who decide on the design and 
implementation of the interventions and how the intervention is to be 
evaluated. Furthermore, he raised the concern that the special epistemic 
status given to RCTs contains strong (explicit or implicit) assumptions 
about what kind of knowledge matters and, therefore, whose knowledge in 
society ought to be given greater weight and attention. In addition, he 
raises the concern that the manner in which the RCT approach has been 
promoted and popularised reflects the exploitation of deep inequities - 
particularly between the Global North and Global South - in monetary, 
institutional and intellectual resources - in order to secure influence over 
public policy decisions. The direct negative effect of these issues is to 
create or compound epistemic inequities. 


In Chapter 11, the authors focus on the development of a new, Made in 
Africa, Transformative Equity evaluation criterion. The authors argue that, 
given the importance of the Organization for Economic Co-operation and 
Development’s - Development Assistance Committee (OECD-DAC) criteria 
in guiding the evaluation process, an explicit criterion is needed to promote 
greater equity in and through evaluations. The authors discuss the process 
of developing the evaluation criterion. The pilot process of integrating the 
criterion into the terms of reference and implementation process of selected 
government agencies is included. Further, the chapter outlines lessons 
learned in this process and reflections on how monitoring and evaluation 
approaches can be extensively transformed to promote Transformative 
Equity in society. 


In Chapter 12, Fish, Mkhize and Masvaure conclude by highlighting the 
key points from the book. 
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E introduction 


Today’s world is littered with various forms of inequalities. Communities in the 
Global South must confront these inequalities daily, and they have become a 
part of their lives. Discrimination and deprivation caused by inequalities have 
brought social and economic violence to the Global South. There have been 
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several approaches to addressing inequalities, but they yielded minimal results. 
What is inconspicuous are the solutions to address these inequalities - the 
solutions have been far and short. In some quarters, there is resignation that 
inequalities are a permanent feature of life, and communities have accepted 
that they have limited powers to change their situation. Inequalities are driven 
by several factors, including gender, age, origin, ethnicity, disability, sexual 
orientation, class, culture and religion. In addition, there are inequalities 
between countries, as well as Global North and Global South power 
asymmetries. These inequalities have shaped the approach to international 
development and its accompanying evaluation practices. 


The World Health Organization (WHO) (2023) defines equity as ‘the 
absence of avoidable or remediable differences among groups of people, 
whether those groups are defined socially, economically, demographically or 
geographically’. Equity strives to eradicate unjust and preventable situations 
that strip individuals of their rights. Consequently, inequalities typically 
emerge when specific demographic groups are unjustly denied access to 
fundamental resources that are accessible to other groups. A disparity is 
considered ‘unfair’ or ‘unjust’ when it arises from societal conditions rather 
than biological factors (Singh, Chandurkar & Dutt 2017, p. 99). Equitable 
evaluation contends that conducting evaluation practices with an equity 
approach is more powerful, as evaluation is used to advance equity. It 
emphasises that the influence of context, culture, history and beliefs shapes 
the character of assessments, particularly within the multifaceted African 
context. Moreover, equitable evaluation has the potential to empower the 
marginalised, amplify the voices of the silenced and bring visibility to those 
who have been rendered invisible (Krenn 2021, p. i). 


Evidence from the literature shows that inequality is prevalent in the 
African continent, hence the need to focus on evaluative solutions that 
address the structural and socio-economic issues that contribute to the 
different forms of inequality such as economic, political and social inequality. 
Despite a plethora of develooment interventions on the African continent, 
a large proportion of the population still lacks access to basic goods and 
services for survival. The effectiveness of developmental programmes in 
sub-Saharan Africa has been elusive, to the extent that minimal inroads 
have been made in addressing key challenges such as poverty, inequality 
and the effects of climate change. One is forced to ask the question: Why 
is it that many people in Africa are without clean water? Why is it that the 
majority are without food, medicine, education or a political voice? Why is 
it that the majority suffer from human rights abuses and poverty? The 
realities cut far deeper than just being poor. 


This book seeks to stimulate conversations among development 
evaluators, commissioners of evaluations and development programme 


Chapter 1 


decision-makers about the role of evaluation in addressing inequality and 
fostering an equitable society in Africa. The book chapters explore the 
following: (1) Take stock of what we know about inequality - what is 
inequality in the African context and how does it affect the lives of the 
citizens of African countries? (2) What does equitable evaluation mean? In 
what way can the concept of equitable evaluation be adopted in evaluation 
practice? (3) What lessons can be learnt from evaluations of interventions 
that address inequality at various (sectoral, programmatic, project) levels? 
(4) What epistemological transformation in evaluation practice is needed 
to achieve an equitable society? (5) How have issues of inequality 
manifested within evaluation practice through organisations, institutions 
and international development? 


This chapter will lay the groundwork for the rest of the book, defining 
what inequality is and the role that evaluation can play in addressing 
inequality. 


E What is inequality/inequity? 


While inequity and inequality are often used interchangeably, they are in 
fact distinguishable. Inequity is regarded as a form of inequality, as it 
involves the unequal distribution of resources. In this case, the unequal 
distribution of resources is based on an unjust power imbalance, often 
because of injustices against historically excluded or marginalised groups 
of people (Hasty, Lewis & Snipes 2023). Inequities generally emerge when 
specific population groups are unfairly deprived of basic resources that are 
available to other groups (Stewart 2013). Social inequalities lead to inequity 
when the groups in charge of distribution allocate resources in ways that 
further oppress marginalised groups. Social inequalities are inequalities 
resulting from people’s backgrounds and the way in which opportunities 
were afforded or limited because of systems of capitalism, colonialism, 
racism, sexism, classism and other forms of oppression (Hasty et al. 2023). 


Theorists have identified four levels of social inequalities as depicted in 
Figure 1.1. 


Interpersonal inequalities are power imbalances that originate from 
personal biases, occur daily and demonstrate and naturalise inequalities 
that exist at institutional and systemic levels. Income inequality is an 
example of this. Furthermore, there are institutional inequalities that arise 
from the implicitly or explicitly biased policies and practices of the 
organisations in society which perpetuate oppression, including educational 
institutions, governments and companies. This kind of inequality is often 
imperceptible and viewed as status quo. Structural inequalities are the 
third type of social inequality, and they are based on the accumulated 
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Structural: 
Global, societal and historical 
levels of oppression 


Interpersonal: 
Power imbalances between 
people that reify social 
inequalities 


Source: Hasty et al. (2023). 
FIGURE 1.1: Levels of social inequalities. 


effects of institutional decisions across societies and history, making them 
pervasive, global and particularly difficult to change. Structural inequalities 
can reinforce individual biases. Lastly, there are systemic inequalities that 
are the convergence of interpersonal, institutional and structural inequalities. 
Systemic inequalities are often depicted by ‘isms’, such as racism, classism 
and sexism (Hasty et al. 2023). Figure 1.1 shows that while social inequalities 
may often be viewed as different phenomena, they are in fact interconnected 
and exist in many different interactions between people and institutions. 


There are also other dimensions of inequality, including economic 
inequality (income inequality, wealth and consumption), gender inequality, 
educational inequality and health inequality. Gender inequality is when a 
person is discriminated against because of their sex or gender, including 
females and people who identify as transgender and non-binary. Gender 
inequality often manifests as unequal treatment in the home, in relationships, 
at work, in communities and in society, including having limited or no 
opportunities to learn, to make money and to hold leadership positions, and 
at its worst, in gender-based violence (Oxfam 2019; Silva & Klasen 2021). 
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Gender inequity, on the other hand, refers to a lack of fairness or justice 
for females in all soheres of society including the economy and labour 
market, education, health care, etc. It is understood that creating equity 
across genders requires the temporary use of special measures 
and treatment to compensate for historical or systemic bias, discrimination 
and disadvantage that result from gender stereotypes, roles, norms and 
differences between the sexes (Shang 2022). Similarly, health inequities 
are the unjust differences in the health of people from different social 
groups, which are linked to various forms of disadvantage, such as poverty, 
discrimination and lack of access to services or goods. Health inequities 
stem from social injustices that result in some population groups being 
more exposed to health risks, and becoming more vulnerable to poor 
health than other groups (World Health Organization [WHO] 2023). 


There needs to be a paradigm shift away from striving for equality, which 
involves giving all people the exact same resources regardless of their 
needs or already existing opportunities or resources and moving towards 
achieving equity. Equity recognises that people have different circumstances 
and needs, and thus, different resources and opportunities should be 
allocated to help these population groups thrive in order to achieve equal 
outcomes for all (United Way 2023). The premise of this book is that instead 
of promoting the allocation of equal resources to people (a one-size-fits-all 
approach or equality), evaluation practice can be used to determine where 
resources and opportunities need to be allocated most to give all citizens 
the same opportunity to thrive (equitable distribution or access). 


E Equity, diversity, inclusion and inequality? 


While the concepts of diversity, equity and inclusion are often viewed as 
synonymous with one another, they are, in fact, different but interconnected. 
Diversity refers to the presence of any features or differences that set 
groups of people apart from one another within a particular area, including 
race, gender, socio-economic status, language, culture, sexual orientation, 
nationality, religion, (dis)ability, age, etc. (Tan 2019). Inclusion, on the 
other hand, is the intentional act or practise of involving, integrating and 
enabling the participation of people from different backgrounds, with 
different identities, in an activity, place, institution or sphere of society 
(Servaes, Choudhury & Parikh 2022). It is the value placed on the differences 
between people that leads to their inclusion. While equality focuses on the 
equal treatment of people despite their diversity, equity recognises that 
there are existing advantages and barriers in place in society that separate 
groups of people, creating imbalances in needs, resources and the 
availability of opportunities for the more marginalised groups. These 
imbalances and barriers within the social systems of society warrant 


Inequality through the evaluation lens 


equitable solutions. Equity is the approach of providing fair access to 
resources, support and opportunities appropriate to the needs of certain 
groups of people and necessary to facilitate equal outcomes (MacEachern 
2019; Servaes et al. 2022; Tan 2019). 


To understand the differences and similarities between inequality and 
inequity, it is important to acknowledge their roots in systemic and systematic 
oppression and power dynamics. Systematic oppression is defined as the 
intentional mistreatment of certain groups of people. Inequalities experienced 
by people from different backgrounds, with different identities, stem from 
oppressive and unequal social systems such as racism, sexism, classism, 
patriarchy, etc. Discrimination and bias (whether covert or overt), on the 
other hand, stem from systemic oppression, which is the way in which social, 
economic and political inequalities are normalised and perpetuated in 
institutions, government and society through laws, policies and practice. The 
concepts of inequality and inequity should also be understood based on 
how they relate to issues of power. Power means the ability to exert control, 
authority or influence over others. Groups of people with more power also 
have more agency, meaning that they have a greater capacity to act and 
make decisions. The agency is also heavily affected by social groupings such 
as race, gender and class (Hasty et al. 2023). 


Racism, for example, results from the intersection between power and 
racial prejudice. Racism is then perpetuated through interpersonal, 
institutional and systemic practices. Racism is when race is used to create 
and justify a social hierarchy and a system of power that privileges and leads 
to the advancement of certain groups of people at the expense of others. 
Racial inequality and inequity are pervasive globally because of racism, 
which is in fact a system based on race, which is a social construct or social 
categorisation that was used to create false scientific superiority and 
authority of certain race groups. Similarly, gender inequality and inequity 
stem from the intersection between gender prejudice and patriarchy. 
Patriarchy is a system of social inequality based on gender, where males are 
perceived to have power and characteristics associated with femininity are 
granted less value. Patriarchy is related to the history of males and contexts 
in which males hold more political, social and economic power and privilege. 
Similarly, income inequality stems from socially constructed class systems 
(Hasty et al. 2023). It is also important to understand the concept of 
intersectionality, in which these different forms of social categorisation 
overlap and exacerbate bias and discrimination. Intersectionality refers to 
the concept that combinations of social identity can have a further impact 
on oppression or privilege. ‘Gender inequality and racial discrimination are 
examples where discrimination experienced because of gender, such as 
discrimination against females, can be directly related, encouraged, and 
shaped by race or ethnicity’ (MacEachern 2019, p. 2). 
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E What are the root causes of inequality 
in the Global South? 


The root causes of inequity in the Global South can be traced back to 
colonialism in the early 1800s. Slavery, genocide, land dispossession and 
racial capitalism were all elements of this history influenced by the interests 
and motives of the Global North. It also had profound impacts on culture 
and value systems, as well as on the way development is viewed and 
pursued. Colonisation was driven by the extraction, exploitation and 
plundering of natural resources from the Global South. Europeans pillaged 
and plundered the colonised communities and nations for the benefit of 
Global North economies, with little being left for the natives of the Global 
South. This period in history has resulted in the affluence and development 
of the Global North today. The pillaging and plunder were disguised as part 
of civilising the Global South; however, it was also driven by racism, human 
rights violations and improvisation. To a larger extent, the current 
development challenges in the Global South are rooted in colonial history. 


During colonialism, fundamental structures and systems of organisation 
in social, political and economic sectors, as well as the role of the state and 
the relationship between citizens and systems of government, were altered 
(Taiwo 2010). Further, colonialism disrupted indigenous ways of life, where 
females contributed to economic structures, whereas today, females lack 
status in every aspect of life. Colonial Christianity brought ideologies and 
belief systems that displayed females’ role in society as being reduced to 
second-class status and removed from the existing religious, political and 
economic systems (Acemoglu & Robinson 2010). In the same way, females 
did not share power with males, while males held second-class status in 
relation to European colonisers. This way of life has taken root as a 
consequence of African people living in colonialism, putting to nought the 
idea that colonisation civilised the Global South. The failure to recognise 
and acknowledge the factual realities of colonial history and how exploitative 
structures continue, how inequity is the invisible and unrecognised privilege 
of history, is important in order to understand how global systems and 
societies today continue to perpetuate inequity through international 
development. 


Colonial inequalities still exist today, however, in different forms and 
terminology. Global North continues to push neo-colonial policies that 
perpetuate the continuation of exploitative economic relations, political 
systems and unrecognised privileges of Global North in geopolitical 
relations long after the attainment of political independence in Global 
South countries (Andrews 2021; Frankema 2005; Nkrumah & Nkrumah 
1965). Neo-colonialism manifests through the control of natural resources 
because of racist policy agendas and the centring of European knowledge 
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in development. The Global North is positioned as the ‘knower’ and 
authority on development, governing global policy agendas and priorities 
through the provision of aid. The perpetual nature of colonialism and neo- 
colonialism displaces endogenous agendas with Global North agendas 
sustained by colonial asymmetrical and intersubjective relations between 
the coloniser and colonised (Ndlovu-Gatsheni 2015). The Global North 
maintains the power of ‘who sets the agenda?’. The power to decide what 
is relevant and what is not, what can and cannot be discussed and what 
matters and what does not (Batliwala 2002). In addition, as during the 
colonial period, the Global South economies are largely extractive, that is, 
driven by primary industries that feed raw materials to the countries in the 
Global North. 


Perpetuation of colonialism manifests itself in asymmetrical power 
imbalances in the global political arena, Western aid and development 
philosophies. Agenda-setting processes are definitively shaped by Global 
North donors’ policies (Bradley 2008). In addition, the Global North 
controls the Global South agenda-setting processes, development values 
and priorities. Global North state agencies and donors’ influence on 
Global South recipients’ agenda-setting through the funding relationship 
between donors and recipients, donors and states, and its impact on 
agenda-setting in contemporary North-South relations. Unlike other 
forms of power, agenda-setting operates covertly in development 
processes, influencing the critical inquiry into what development is 
(Capella 2016). As a result of asymmetrical power imbalances between 
North and South, agenda-setting power continues to perpetuate inequality 
and inequity in development agendas, for example, how problems are 
identified and defined in pursuit of development and the accompanying 
interventions. 


E Why is it important to deal with inequity? 


Notably, in the early 2000s, there were policy shifts by the traditional 
Global North funders of development in the Global South. The policy shifts 
were largely informed by the poor impact of development programmes in 
the Global South (Pallas & Urpelainen 2013). Consequently, development 
in the Global South took on a new focus with an increased focus on selected 
topics of interest that are defined in the Global North. Within these 
new topics, there was also an increased need to evaluate and measure 
immediate and short-term outcomes of the various funded development 
interventions. Although the need for evaluations of interventions was noble, 
it was defined by Global North values, paradigms, approaches and 
methodologies. The implication was that evaluation became a tool that 
validates and perpetuates the colonial ideas of the Global North. 
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Countries in the Global South inherited disempowered economic 
structures and high inequality levels from the colonial era and have 
remained high since independence (Stewart 2005). Recent growth and 
development patterns in many countries are increasingly defined by 
inequities across economic, human, social, political and intergenerational 
dimensions. Despite a plethora of development interventions on the 
African continent, a large proportion of the population lacks access to 
basic goods and services for survival. Development interventions in sub- 
Saharan Africa have been elusive, to the extent that minimal inroads 
have been made in addressing key challenges such as poverty, inequality 
and, currently, climate change (Masvaure & Motlanthe 2022). Inequity 
threatens sustainable development and human rights, perpetuates 
intergenerational poverty and limits communities and personal quality 
of life, dignity and self-worth. Thus, addressing and locating inequities 
in development is indisputable for development to make real 
contributions to improving well-being and combating inequities. 
Addressing inequity broadens and deepens development efforts’ 
impacts at multiple levels. It challenges the pathways and processes of 
development, necessitating the unbundling of asymmetrical power 
relationships and dynamics developed through centuries of colonialism 
and neo-colonialism. Further, it encourages disadvantaged social 
groups, communities and states. It compels us to confront the perpetual 
cycle of injustice. 


Inequality is a manufactured reality and a conscious decision, 
recognising that the decision not to tackle it in the face of increasing 
poverty, climate impacts, health inequities, unemployment and gender 
disparities has adverse impacts on inducing avoidable human suffering. 
The danger of the current ‘silent emergence’ of inequities as an acceptable 
societal condition and a distinct feature of modern societies negates the 
historical events that have led to the rising and perpetual impacts of 
inequity. Instead, a critical lens should be applied to development 
trajectories and agendas to achieve systemic change. Acknowledging the 
need for change requires deep inquiry into what limits us from tackling it. 
It also requires locating responsibility within the broader system. 
Fundamentally, addressing inequality and injustice is a struggle towards 
undoing colonial and neo-colonial exclusion and subjugation of people 
and communities in the Global South. This requires not only strengthening 
development efforts to become appropriately designed but also 
responsive to denied equity and justice in all sectors of modern society. 
Ultimately, it necessitates centring disadvantaged groups’ lived histories 
and experiences at the centre of development agendas. This avoids the 
silent emergence of inequity and ahistorical narratives of neo-colonial 
development. 
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E Post-colonial development and inequality 


Several concerted efforts are targeted at addressing the various types of 
inequalities in the Global South. The efforts to address inequality in the 
Global South are rooted in the colonial power asymmetries where the 
Global North holds the decision-making power on funding and prioritisation 
of development. This approach mirrors how colonisation took place - the 
mantra was to spread civilisation to uncivilised countries such as those in 
the Global South. The Global North was purported to be civilised and felt 
that they had the responsibility to spread civilisation (Moloney 2020). 
However, the results of the process of spreading civilisation are shrouded 
in brutality, deprivation and the impoverishment of the Global South. 
During the colonial period, the Western powers propagated a specific 
interpretation of their own values (civilisation values) to justify colonisation. 


Thought leadership for the current approaches to international 
development is held by the Global North. There is also unchallenged 
leadership of the Global North on development priorities in the Global 
South (Gosovic 2000). The north-south divide inequalities have not been 
challenged despite the increasing pressure for addressing such inequalities. 
The power asymmetries also define what gets funded. In the Global South, 
ideas, policies and intentions are spelled out, but if these ideas are not in 
tandem with the Global North agenda, they are starved of funding (Prashad 
2014). In addition, international development divides countries according 
to ‘developed’ and ‘underdeveloped’, which closely mirrors the colonial 
division of ‘civilised’ and the ‘uncivilised’. 

Addressing inequality in the Global South has mainly been centred 
around addressing income inequality and, to some extent, wealth inequality. 
Income inequality has been addressed through poverty reduction strategies 
(Odusola 2017). The conceptual thinking around addressing poverty is 
driven by multilateral institutions such as the World Bank and the 
International Monetary Fund through their Global South lending 
mechanisms. Some of the programmes were realised through policy 
instruments such as poverty reduction strategies, structural adjustment 
programmes, Millennium Development Goals and the current sustainable 
development goals (SDGs). What is important is that these policies 
proffered the blueprint for the Global South to progress to full economic 
growth, and that will result in ending poverty and inequality. Except for a 
few countries that charted their own way separate from the Global North’s 
interventions, very little has been achieved when it comes to moving 
countries from being regarded as underdeveloped to developed. The 
international economic order is roughly still the same and exudes the global 
income patterns that were established in the early 19th century (Freistein & 
Mahlert 2016). 
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Redistributive policies have also been promoted in the Global South as a 
way of addressing inequality. These policies were in three parts: wealth 
redistribution, income redistribution and provision of public goods. 
Although there has been progress made in addressing specific inequalities 
through some of these policies, eradicating the associated socio-economic 
inequalities has been elusive. There are also several coordinated 
interventions that seek to address different types of inequalities such as 
health, human rights, gender and education. The lifelines of these 
interventions also depend on and are driven by funding and thought 
leadership from the Global North. Limited success has been achieved with 
these interventions, and evaluations have been guilty of paying too much 
attention to accountability instead of the impact of these programmes 
(Ebrahim 2005; Mayne 2017). The next section will present how evaluation 
can play a critical role in building a just society in the Global South. 


E Contribution of evaluation to addressing 
inequality 


Inequality as a phenomenon and the scholarship around it are not new. The 
chasm between the ‘haves’ and ‘have-nots’ and the vast differences in the 
economies, well-being and livelihoods of developed and developing countries 
have long existed and are well-documented in the fields of economics, health 
and the social sciences, to name a few. This is exemplified by the quantum of 
academic and non-academic publications produced, for example, Stiglitz 
(2012), Piketty (2014) and Milanovic (2011). Conceptually, there are various 
definitions of inequality, for example, income inequality, wealth inequality 
and structural inequality. These disparities in the income, wealth, education, 
food security, access to health care, safety, security, housing and general 
well-being of individuals (to name a few) are related to structural inequalities 
that are produced by the othering and unequal treatment of categories of 
individuals, which is deepened by the institutionalisation of these patterns in 
the systems and structures of society. The development discourse is replete 
with narratives around the need to address inequality and accelerate 
equitable development, and global and national policies are awash with 
intentions to do so. 


The continued existence of ‘developed’ and ‘developing’ countries (or 
richer and poorer nations) is indicative of the notion that development is 
rarely equitable and speaks volumes of the laudable intentions over the last 
few decades of the domination of modernisation, industrial development 
and economic growth that have not been able to produce equal societies 
anywhere in the world. It is widely acknowledged that those who own the 
means of production have power over those who do not - creating and, 
at times, deepening inequalities (Peters 2004). However, international 
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institutions driving the international development agenda, for example, the 
World Bank, have intentionally moved towards addressing inequality 
through shifting towards the ‘shared inequality’ approach (World Bank 
2015). The global movement to highlight the importance of prioritising an 
intentional approach to tackling inequality both within and among countries 
is perhaps most elucidated in the SDGs (United Nations General Assembly 
2014). In this regard, inequality has become a significant thematic area 
within policy and development debates on the global stage. 


Evaluation is a well-established profession with an expanding worldwide 
standing and an abundance of organisations, conferences and textbooks 
(Hall 2020). The evaluation field is characterised by the idea that the 
production of evidence through the systematic collection and analysis of 
data, set apart from research by the value judgements that are invoked on 
the policies programmes and projects under scrutiny, is a public good that 
is conducted in service of addressing the manifestations of inequality in 
society. Evaluations, therefore, are generally conducted on programmes 
that aim to address development problems in sectors as varied as education, 
health, climate change, economic development, social protection and so 
forth. However, what is concerning is that for a long time, the field or 
practice of monitoring and evaluation, particularly evaluation, has had little 
to offer in addressing inequality through findings produced, as the greatest 
challenge is the utilisation of evaluation findings to either improve policies 
or programming. Evaluations also typically focus on single programmes or 
interventions, ignoring systemic issues and the complexity of development 
challenges. Evaluations could even do harm (Shanker & Maikuri 2022). 


Evaluations, in essence, are designed to answer questions related to 
evaluation objectives related to a particular policy or programme that are 
usually of concern to the evaluator or funder rather than the evaluand or 
beneficiaries. The question, therefore, is perhaps not how evaluation can 
contribute to addressing inequality but whether - in its current form - it 
can do so. In the current discourse on the transformation of the evaluation 
sector (both in the conducting of evaluation and the kinds of policies, 
programmes and interventions it concerns itself with), the contribution of 
evaluation to addressing inequality is in question (Van den Berg, Magro & 
Mulder 2019). The idea that better policy- and decision-making for 
contributing to accelerated development results is dependent on the 
policies and programmes that are being evaluated, and if these are 
ensconced in inequality, evaluations alone will not be successful in 
addressing inequality. Systems transformation, therefore - which requires 
the transformation of the structures within which development happens - 
and the evaluations that are conducted within these systems of development 
both require attention. The debate on the transformation of evaluation is 
also linked to the growing discourse on the decolonisation of evaluation 
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and the concern in the Global South broadly, but in Africa in particular 
(where inequality indicators are at their worst), that colonial power matrices 
that continue to exist in, for example, development financing, continue to 
affect the evaluation sector. It is argued that the generally accepted 
evaluation methods, theories and approaches that were designed for 
application and relevance to the Global North, more so the accountability 
concerns of the international finance sector, continue to dominate 
evaluation practice in the Global South (Abrahams, Masvaure & Morkel 
2022). The extent to which evaluation could reduce inequality therefore 
remains a pipe dream unless it is fundamentally transformed itself, and if it 
addresses the transformation of the systems that produce and reproduce 
inequalities. 


However, over the past years, inequality began to surface in the discipline 
and practice of evaluation. For example, Rist, Martin and Fernandez wrote 
a book entitled Poverty, Inequality, and Evaluation: Changing Perspectives 
(2015). Additionally, there is a burgeoning ‘justice for all’ movement in the 
discipline of evaluation that has been seen in the United States of America 
(USA), where it strives to bring about change in the areas of social, 
economic and political inequalities (Hood et al. 2015). Generally, such 
evaluations employ culturally-responsive paradigms and methods and are 
known as social justice-oriented evaluation approaches (Mertens & Wilson 
2012). On the contrary, the question of whether evaluation could be a 
catalyst for addressing inequality remains utopian. 


However, what is known for a fact is that the evaluation of interventions 
now needs to put emphasis on producing evidence that informs how such 
interventions are equally distributing the gains across all in a way that 
reduces inequalities (Agrawal & Rao 2015). In that regard, Bamberger and 
Segone (2011) argue that equity-focused evaluations need to offer inquiries 
regarding what works and what does not contribute to decreasing 
inequality, along with the intended and unintended consequences for the 
worst-off groups, as well as the disparities that exist between the better- 
off, average and worst-off groups, permitting government officials (in the 
context of the evaluation of state-driven public good programmes) to 
integrate the results, recommendations and lessons into decision-making 
processes in the shortest possible time. 


The need for intentional action in addressing deep-seated causes of 
inequality and consequences at individual and country levels is now more 
urgent than ever. Picciotto (2015) calls for the reformulation and rethinking 
of evaluation in the development arena. The search for a solution to 
inequality should therefore be found within the systemic configuration of 
world order and at the country level. According to Fernandez et al. (2015), 
the National Development Plan (NDP) is the plan of action under which the 
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majority of countries that are developing have organised their economic 
and social policies. The NDPs of countries like South Africa, Lesotho, 
Kenya and Namibia frequently involve poverty and inequality reduction. 
Additionally, in the case of South Africa, the Constitution is fundamentally 
anchored on principles of equality and equal opportunities for its citizens. 
Furthermore, it is backed by various legislative frameworks that were put 
in place to bring about equality and equity that were a result of colonisation, 
segregation and the apartheid system. However, despite these laudable 
policy, legislative and institutional mechanisms, the country is still far from 
reaching the level of equality that is articulated in the Constitution. 


The concern is whether public good delivery interventions themselves are 
successful in prioritising addressing inequality and whether programmes to 
address inequality are actually being implemented. The question is, how can 
we evaluate what has not been implemented? If programmes are embedded 
in a context of systemic inequality, and if programmes continue to be 
focused on single problems, denying systemic conditions and approaching 
development from the perspective of single interventions, then how can 
evaluations (if focused on evaluating such programmes) address inequality? 
Evaluation will not help us answer the critical questions if the critical problems 
remain unattended to. Evaluations could be an entry point for informing the 
government to distribute the economic gains equally across all the people 
regardless of race, ethnicity, location and religion if the programmes 
themselves are addressing these issues systemically. 


Agrawal and Rao (2015) argue that the strategic planning of the 
Government of India placed a sense of urgency on addressing inequality 
than before. Programmes of national commitments focused on bridging 
the gap between the haves and have-nots; programmes such as the 
Mahatma Gandhi Rural Employment Act, the School Feeding Scheme and 
Health for All (Agrawal & Rao 2015). 


Considering the issue raised about NDPs, Fernandez et al. (2015) argue 
that given the complexity thereof covering all sectors of a respective 
country - and also the inclusion of all stakeholders - it is a challenge to 
summarise micro viewpoints into macro perspective and make the plan a 
utilisable document. The macro viewpoint further complicates grappling 
with the attribution problem during evaluations of national commitments 
or interests. The question, therefore, is how can we know that the NDP 
made an impact, notably in lowering inequality and unfairness, in the 
absence of a counterfactual? (Fernandez et al. 2015). This provides a key 
takeaway for evaluators evaluating NDPs and wanting to use findings to 
address the inequality. 


Now more than ever, evaluation can contribute to addressing inequality 
and inequities in a variety of ways. For example, the SDGs and new 
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approaches to providing development financing call upon countries to 
develop country-led, focused and owned evaluation systems. Evaluating 
social justice and equity programmes, focusing on prioritising evaluations 
that address the issues of inequality. Prioritising these in National Evaluation 
Plans (NEPs) could be the critical levers in creating the transformation that 
is needed in order for evaluation to address inequality. 


E Conclusion 


In conclusion, some of the issues that evaluators are grappling with provide 
some semblance of other actions that can provide leverage for 
transformational shifts in the evaluation towards accelerated equitable 
development. Firstly, conducting and ultimately ensuring the utilisation of 
evaluation results make decision-makers answerable for the fair distribution 
of opportunities and resources. Evaluation experts can verify that resources 
are allocated fairly and that interventions are effective in combating the 
fundamental root causes of inequality by monitoring the impact of policies 
and programmes on different groups of people. However, this is affected 
by a world order that is marked by political disintegration, interdependent 
economies and multilateral organisations contaminated by serious 
democratic deficiencies compounded by a lack of strong political 
leadership; this makes it almost impossible to say that coherent and timely 
public decisions will be made (Picciotto 2015). There are, therefore, 
possibilities for evaluations to provide solutions. Picciotto argues that ‘L...] 
there is a need for dispassionate, independent, and rigorous assessments 
of public policies and programmes’ (Picciotto 2015, p. 255). One key 
question is whether evaluation as a practice and discipline is ready to 
provide solutions to complex and weak public sector systems? Perhaps to 
answer this question, countries are putting infrastructure for evaluation(s) 
to navigate such complexities. Countries like South Africa, Benin, Colombia, 
Mexico and Uganda have national evaluation systems (Goldman et al. 2019). 
Additionally, Benin, South African, Uganda and Uruguay have national 
evaluation policies and plans upon which evaluation functions are structured 
(Genesis Analytics 2017, p. 85). 


Secondly, inequalities in having access to resources, opportunities and 
outcomes can be found and assessed through different evaluations. In that 
regard, evaluators can shed light on how certain categories of people may 
be disadvantaged compared to others by examining data and comparing 
outcomes across groups. By gaining insight into the reasons behind (and 
the scope of) inequality, governments and other development actors are 
able to prioritise resources and solutions channelled towards those who 
are not receiving equal resources. For example, Uganda’s parliament 
recommended that the Youth Livelihood Programme be evaluated, and in 
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a report, the Auditor-General alleged that the government’s Youth 
Livelihood Programme was failing to provide young people with the start- 
up funding and skills they needed (Chirau et al. 2022, p. 112). This exerted 
pressure on the government to channel resources where they are needed, 
for example, capacity-building and funding for enterprises. 


Thirdly, findings from evaluations may be utilised to justify changes in 
policy that will confront inequalities. Evaluators are able to assist 
governments in making well-informed choices regarding which policies 
and programmes (Lederman 2012) should be promoted by proving the 
successful outcomes thereof that target inequality. 
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E introduction 


The Made in Africa Evaluation (MAE) approach has emerged and challenged 
Euro-American evaluation frameworks that largely dominate the assessment 
of development interventions in Africa (Chilisa 2019; Gugerty, Mitchell & 
Santamarina 2021). Euro-American evaluation frameworks have been 
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evidenced to use a predominantly top-down extractive approach that 
serves the reporting needs and interests of donors but often fails to capture 
the actual impact of interventions and leaves programme participants 
voiceless, objectified and disempowered (Khumalo 2022; Masvaure & 
Motlanthe 2022). While positive changes to international donor 
requirements and evaluation methods are occurring, the MAE approach 
attempts to further contextualise evaluative practice by considering how 
African philosophies, cultures, belief systems and ways of knowing can be 
incorporated into evaluation frameworks to measure success as experienced 
by programme participants (Chilisa 2019; Omosa et al. 2021). This approach 
promotes the use of locally relevant data-collection methods, such as 
storytelling, to align evaluations to the expectations, needs and realities of 
local programme participants (Omosa et al. 2021). Importantly, the MAE 
approach is defined by a set of guiding principles that aim to promote 
social justice and equity in evaluation. These principles include addressing 
power inequalities among stakeholders, embracing indigenous worldviews 
and knowledge systems, and the inclusion of a multiplicity of voices in the 
evaluation process (Chilisa & Mertens 2021). The MAE approach therefore 
supports rigorous and ethical evaluation practices. 


While the MAE approach has a set of well-defined principles, there has 
been limited research on how these principles are operationalised in a way 
that satisfies the needs and expectations of all stakeholders within the 
evaluation ecosystem. The evaluation ecosystem refers to all the people, 
processes, types of knowledge, information and skills that are involved ina 
programme evaluation. The main challenge is to ensure that the practice of 
MAE serves its purpose of social justice and equity while negotiating the 
needs and interests of the different stakeholders. 


Critically, this positions the evaluator at the centre of operationalising an 
MAE approach. Yet, we know little about what is required in practice by 
evaluators to challenge the approach set by donors, negotiate whose success 
to measure and how to measure it and ensure the representation of a plurality 
of realities and experiences (Chilisa 2019; Knumalo 2022; Mbava & Dahler- 
Larsen 2019; Omosa et al. 2021; Smith 2021). This is especially relevant when 
evaluators are required to conduct an evaluation on existing programmes, 
where MAE may be at odds with the original programme design. 


This study thus explores the overarching question: What does it mean, in 
practice, for evaluators to stay true to MAE’s purpose of social justice and 
equity? To answer this question, this research investigates three key aspects 
of evaluators’ work: How do evaluators negotiate tensions and power 
dynamics in the evaluation ecosystem? How can evaluators stay true to the 
lived experiences of programme participants? How can evaluators best 
ensure inclusivity and representation in the evaluation processes? 
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Our study was set around two youth development programmes operated 
by a non-governmental organisation (NGO) in rural KwaZulu-Natal, 
South Africa. Using a participatory approach and a range of qualitative 
research methods, we worked with key stakeholders to co-create an 
evaluation framework that captured the lived experiences of the programme 
participants. To gain insights into the process, we engaged in observations 
and conducted sixteen semi-structured interviews and four focus groups 
with key stakeholders, including programme participants, NGO employees, 
donors and community tribal authorities. 


In brief, we found that when applying an MAE approach, the role of the 
evaluator extends beyond its traditional scope to include the active 
facilitation of collaboration between key stakeholders. If guided and 
supported by the evaluator, donors are often willing to adapt their current 
evaluation frameworks to better suit the African context. We also found 
that to stay true to the lived experience of programme participants, 
evaluators need to recognise the subjective and relational nature of success, 
looking beyond individuals to an impact ecosystem that includes family 
members, the broader community and the environment. Finally, as there is 
no one-size-fits-all approach, community members need to be included in 
the co-creation of an evaluation framework that best aligns with their 
values, purpose and realities. The process of co-creation requires evaluators 
to hold space to foster trust and build relationships - both critical elements 
for ensuring success in the evaluation design process. 


Our study contributes to a growing body of evaluation research that is 
situated in Africa, for Africa, by providing insights into how an MAE 
approach can be implemented in practice. It further provides guidance to 
evaluators on how to implement Africa-rooted evaluations that promote 
social justice and equity while continuing to recognise the diversity of 
needs, expectations and realities in the evaluation ecosystem. 


E Background 
Western influence on evaluative practice 


The African development sector has historically been dominated by 
Western worldviews, assumptions and interests (Khumalo 2022; Said 1993; 
Smith 2021). Local NGOs involved in development efforts are largely reliant 
on donor funding, which comes from myriad sources such as international 
aid agencies, multilateral institutions (e.g. World Bank), foreign governments 
and local corporations (Hayes, Introna & Kelly 2018). The reliance on donor 
funding has served to reinforce unequal power dynamics, often allowing 
Euro-American actors to dictate development priorities and practices. 
These actors tend to focus their interventions on a single core problem and 
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use tools such as logic frames, result chains and impact pathways to work 
backwards from a predetermined outcome to identify a set of enabling 
activities (Belcher & Palenberg 2018; Chilisa 2019; Gugerty et al. 2021). 


Evaluation frameworks used to assess the success of interventions have 
historically been rooted in the paradigms and culture of the West (Said 
1993; Smith 2021). These evaluation frameworks, which structure how 
programme outcomes are defined and how data are collected and analysed, 
are guided by the epistemological view that assumes a causal relationship 
between an intervention and its desired effect (Belcher & Palenberg 2018). 
They are often based on a one-size-fits-all approach, which emphasises 
objective assessment to ensure accountability and replicability of 
interventions (Masvaure & Motlanthe 2022). While this approach might be 
well-suited to the context in which it was created, it often fails to capture 
the complex realities that exist within the African context (Chilisa 2019; 
Mbava & Chapman 2020; Mbava & Dahler-Larsen 2019; Omosa et al. 2021). 


The principles that have traditionally guided evaluations tend to 
disempower local communities (Chilisa 2019; Mbava & Chapman 2020; 
Mbava & Dahler-Larsen 2019; Omosa et al. 2021; Smith 2021). Firstly, donors 
have the habit of defining the problem in a community and assigning 
various measures of success to the intervention without adequate 
participation from other project stakeholders (Hayes et al. 2018; Smith 
2021). Secondly, evaluations are conducted on communities, not with them, 
and results are fed back to the donors for their benefit only (Chilisa 2019; 
Hayes et al. 2018; Omosa et al. 2021). Thirdly, this top-down extractive 
evaluation approach pays no consideration to what the community is 
getting out of the evaluation or what their actual needs and interests are 
(Khumalo 2022). 


In line with these principles, the metrics developed by the donors are 
in line with a return-on-investment approach to measuring impact (Hayes 
et al. 2018). This means that the metrics are predominantly quantitative 
and make use of scientific methods to discover objective truths about 
an intervention and determine what success should look like. In many 
cases, projects are designed to achieve high-impact evaluations rather 
than genuinely improve the situation for the programme participants 
(Hayes et al. 2018). Time and resources are directed towards meeting 
evaluation goals and producing evidence of impact for donors rather than 
assessing value and success from the perspective of those being assisted 
(Khumalo 2022). 


The evaluation practices that favour Western knowledge and value 
systems for data-collection and reporting have been evidenced to 
reinforce the asymmetry of the relationships between donors and local 
communities (Chilisa 2019; Smith 2021). African ways of knowing and 
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relational knowledge-sharing approaches are considered illegitimate (Chilisa 
2019; Smith 2021). As a result, practices such as storytelling, folklore, music 
and dance are excluded from evaluation frameworks. This culminates in an 
evaluation framework that promotes a one-size-fits-all approach that fails 
to benefit both the donors (who are spending money ineffectively) and the 
communities (whose agency is being undermined). 


Rise of Made in Africa Evaluation 


In response to the shortcomings of evaluative practices, new evaluation 
approaches have emerged over time, such as developmental evaluation, 
indigenous evaluation, context-specific evaluation, participatory evaluation 
and culturally-responsive evaluation - to name a few. While each has its 
own set of values, practices and tools, they all seek to be more cognisant 
and responsive to culture and context (Chilisa 2019; Chouinard & Cram 
2019; Patton 2011). 


The MAE approach aligns with and draws from these approaches to a 
large extent, yet it is unique in that it calls for a distinctively African way 
of doing things (Chilisa 2019). The African Evaluation Association (AfrEA) 
has put forward an MAE approach to acknowledge African indigenous 
knowledge systems (IKSs), beliefs and practices (Chilisa 2019; Omosa et al. 
2021). The MAE approach seeks to report on success and failure in a way 
that honours the lived experiences of local African communities (Chilisa 
2019; Mbava & Dahler-Larsen 2019; Omosa et al. 2021). The definition of 
MAE, as refined by Omosa et al. (2021), is: 

[E]valuation that is conducted based on AfrEA standards, using localised 

methods or approaches with the aim of aligning all evaluations to the lifestyles 


and needs of affected African peoples whilst also promoting African values. 
(p. 6) 


The MAE approach challenges current evaluative practices that marginalise 
African data-collection tools, do not acknowledge African diversity and 
leave local stakeholders wondering how exactly the community has 
benefited from programmes (Chilisa et al. 2015). 


The MAE approach is guided by relational ontology and epistemology 
(Chilisa 2019). A relational ontology recognises people’s connection to 
non-material things, such as spirits or sacred places, and thus, an MAE 
approach will include a holistic view of well-being - one that includes the 
community and the environment (Chilisa 2019; Smith 2021). A relational 
epistemology places value on the process of knowledge creation, where 
meaning is stored in the unique languages, practices, rituals and folktales 
that are held by its people (Chilisa 2019). These modes of knowing form the 
basis for the methodological design of an MAE approach, making it an 
approach that is rooted in a relational paradigm. 
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Translating Made in Africa Evaluation principles 
into practice 


While the MAE approach is associated with a set of defined principles, 
these have not yet been translated into a well-documented evaluative 
praxis (Chilisa 2019). Thus, questions remain as to how to go about, in 
practice, negotiating tensions and power dynamics among stakeholders, 
uncovering and incorporating indigenous worldviews and representing a 
multiplicity of voices in the evaluation process. 


This brings to the foreground the role of evaluators, who are likely to 
shape MAE practice as they operationalise the principles (Mbava & 
Chapman 2020; Omosa et al. 2021). A key challenge for evaluators is to 
allow practices and processes to be highly contextualised, steering away 
from a desire to rely solely on technical know-how or to seek a set of 
predetermined toolkits. Research has thus highlighted the need for 
evaluators to have relational and transformative competencies (Rosenberg 
& Kotschy 2020). Relational competencies refer to the evaluator’s ability to 
engage, collaborate, facilitate and build trust with a group of stakeholders, 
while transformative competencies refer to the ability of an evaluator to 
see the need for and to develop alternative evaluation methods and 
processes that suit the context (Rosenberg & Kotschy 2020). 


Importantly, and as highlighted by Chilisa et al. (2015), to advance MAE 
practice, evaluators must grapple with key challenges relating to hierarchy, 
power and representation. Firstly, there is a need to challenge the top- 
down, extractive manner of evaluation that favours the needs and interests 
of donors over the participants (Khumalo 2022). This may not be easy in 
contexts where programmes have been initiated without a transparent and 
upfront agreement on the use of an MAE approach. Secondly, evaluators 
must challenge whose values matter and agree on whose success to 
measure and how to measure it (Khumalo 2022). This is critical as value 
considerations are likely to inform decisions on future programmes. It also 
represents an opportunity to shift power imbalances in knowledge 
generation and use; however, this may be easier said than done in a context 
where certain expectations have long been associated with evaluative 
practices. Thirdly, evaluators must work to ensure the representation of a 
plurality of realities and experiences through their practice. Key to this is 
ensuring that programme participants are involved in defining what 
inclusion means to them and how the representation of voices should be 
organised (Chilisa et al. 2015; Khumalo 2022). 


As with every emerging concept, there is a need to learn from practice how 
to be intentional about MAE without alienating certain stakeholders. To avoid 
MAE becoming a one-size-fits-all approach, it is essential to explore how 
contextualisation happens and how to facilitate stakeholder engagement. 
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Gaining and sharing insights from practice can ensure that MAE reaches 
wider coverage, acceptability, prominence and use in the African context 
(Omosa et al. 2021). 


E Research design 
Research setting 


To explore how an MAE approach can be implemented in a way that satisfies 
the needs and expectations of all stakeholders, and to answer our research 
questions, we embedded ourselves into the evaluation process of an NGO 
involved in two youth development programmes situated in five rural 
communities in Northern KwaZulu-Natal, South Africa. One of the 
programmes provided youth (18-27-years-old) who had completed tertiary 
education with an opportunity to gain work experience, while the other 
provided entrepreneurship and business support to those youth 
(18-35-years-old) who had not attended university but were participating in 
the informal economy. The participants were required to apply for the 
programme and were then selected by the NGO equally across community, 
age and gender. Given the location of the programme, the participants all 
identified as Zulu. It is important to note that this chapter does not detail 
the respective programme evaluations but rather focuses on the process 
and insights gained from designing an MAE framework. 


While the NGO involved had not designed their programmes with an 
MAE approach in mind, they were interested in understanding how the 
principles of MAE could be adapted to facilitate the evaluation of their 
community-led programmes. The NGO was formed in 1992 with a mandate 
to ensure that local communities benefit from the growing eco-tourism 
sector. Their approach emphasised the importance of involving local 
community members in the conceptualisation and implementation of all 
phases of programme development. 


The NGO was reliant on donor funding and was thus occasionally 
required to act in an opportunistic manner to secure this funding. The 
programmes were funded by a few South African corporation, through 
their social investment portfolios. Thus, this research investigated the 
attributes and practices related to evaluations performed with corporate 
donors and does not aim to make any inferences to donors partaking in 
international development activities. 


The five rural communities in which this study took place are situated 
around a large, protected land area. The physical geography of each of the 
communities differs - from mountainous regions to lowland wetlands and 
everything in between. While these communities face many socio-economic 
challenges - low levels of household income, a lack of job opportunities 
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and difficulty accessing basic services - their natural environment is rich 
and wild. Given the value of these wild spaces, there are many public and 
private conservation and eco-tourism initiatives operating in the area that 
provide some opportunities to local communities. 


Community members identify as Zulu, and this comes through in their 
traditional belief systems, values, knowledge systems and practices. At the 
same time, they operate in a world that is increasingly modernised, and this 
is something that they must balance in their everyday lives. Each of the five 
communities is governed by their own leadership structures and have their 
own developmental priorities. 


Research approach 


In line with the MAE approach, we adopted a relational paradigm (Chilisa 
2019) and used a participatory action research approach (Preiser et al. 
2018). The first author of this chapter was the evaluator for the two 
programmes. She embedded herself into the NGO’s regional operations 
team to gain insights into the programmes, the stakeholder dynamics and 
the participant experiences. Rather than working within the constraints of 
an existing evaluation methodology, the evaluator made use of her 
technical, relational and transformative competencies to capture the lived 
experiences of the participants and co-create an evaluation framework 
that could accurately reflect the impact of the programme from the 
participant’s perspective. 


We further incorporated decolonisation practices into our research 
approach. Firstly, we treated research participants as experts (Smith 2021) 
and designed our methods of engagement to enable co-creation. Secondly, 
we spent time building relationships of trust with the research participants 
(Thambinathan & Kinsella 2021). Thirdly, we considered the regular sharing 
of emerging findings with participants to be an integral part of the research 
process (Smith 2021). 


Data-collection and analysis 


In line with a relational paradigm and decolonisation practices, the first 
author first spent time building relationships of trust before conducting 
any research-related activities (Chilisa 2019; Smith 2021). Thus, for the 
first two months, she embedded herself in the activities of the NGO, 
which allowed her to have many informal relationship-building 
engagements with the programme participants and regional NGO 
employees. These engagements helped her to identify appropriate and 
willing candidates to participate in the study. Throughout the research 
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process, the evaluator was supported by two local research assistants, 
who played an influential role in developing her understanding of the 
context and culture and assisted with translations between isiZulu and 
English when required. 


After these first two months of relationship-building and informal 
engagement, we then iterated between data-collection and data analysis, 
progressively refining our focus and analysis. We initiated data-collection 
with semi-structured interviews and observations. A total of sixteen 
individual interviews were conducted. Firstly, eight programme participants 
(P1-8) were invited to participate in the research in order to understand 
their perspectives on the successes and failures of the two programmes. 
The demographics of these eight research participants accurately reflected 
those of the programme as a whole, in terms of gender, community and 
age. Secondly, five NGO employees (N14-19) were interviewed to gain 
insight into the evaluation systems currently utilised by the NGO, how the 
evaluations represented the lived experiences of the programme 
participants and how the employees dealt with the dynamics and tensions 
associated with reporting to donors. These employees were chosen for 
their involvement in both on-the-ground activities and donor reporting 
processes. Finally, three representatives from the donor organisations 
(D11-15) were interviewed to gain insight into their expectations and 
demands with regard to reporting and their overall attitudes towards the 
programmes they funded. 


The interview data were analysed by coding for themes, progressively 
grouping these themes into categories that captured key insights (Charmaz 
2011). Core to our approach, key insights were challenged through a process 
of member check-ins. During this process, the evaluator shared her findings 
and provided the research participants with the opportunity to clarify or 
oppose any misrepresentations or inaccurate portrayals in the research 
data (Chilisa 2019; Smith 2021; Thambinathan & Kinsella 2021). This ensured 
the findings accurately reflected their views. 


A key insight that emerged was the need to review the current 
evaluation frameworks to better reflect how participants experienced 
success. Building on this core insight, we held four focus groups: two 
sessions with programme participants (F9-10) and two sessions with 
community tribal authorities (F20-21). The aim of the first set of focus 
groups was to provide the participants with the opportunity to define 
what should be evaluated on the programme, as well as when, how and 
by whom (Mbava & Chapman 2020). They were held with two groups of 
programme participants (twelve and sixteen from each programme, 
respectively) who had not yet been interviewed and were representative 
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of the programme in terms of gender, community and age. The focus 
groups were conducted in a workshop style to give the participants the 
opportunity to co-design the evaluation strategy used to assess their 
programme. Two local evaluators, who were employed by the NGO, were 
involved in designing and facilitating the workshop. The purpose of their 
involvement was to provide a direct line of communication between 
participants and NGO employees, support capacity-building among 
local evaluators and assist in neutralising any power dynamics present in 
the space. 


The second set of focus groups was held with the tribal authorities of 
two of the five communities involved in the programmes. These two tribal 
authorities had structures similar to the others and were chosen as they 
had been involved with the NGO’s programmes for the longest period of 
time. In line with an MAE approach, we wanted to understand how the 
programmes were perceived by the traditional leaders of the communities 
(Chilisa 2019). We also wanted to give them the opportunity to define the 
desired content, format and regularity of the NGO’s programme reports. 
These engagements were facilitated by the evaluator with the support of 
the local research assistants. 


After each focus group, the evaluator and supporting team debriefed 
by reflecting on key takeaways, sharing notes and reviewing the evaluation 
frameworks produced by the focus group participants during the session. 
These findings were summarised in a report that included a revised 
evaluation framework that had been co-created with the programme 
participants and tribal authorities. The local evaluators played an active 
role in analysing the data and presenting the resulting framework to the 
NGO management team. 


Throughout the research process, the evaluator recorded observations 
about what was happening in the field and engaged in critical reflexivity 
to constantly examine her epistemological assumptions and situatedness 
in the context (Hart & Whatman 1998; Smith 2021; Thambinathan & 
Kinsella 2021). Observations were made during both formal and informal 
engagements with the programme participants, as well as during on-site 
meetings with regional NGO employees. They assisted the evaluator in 
making sense of the emerging findings and thinking about the next steps 
of the research process. In practising critical reflexivity, the evaluator 
made an active effort to reflect on her positionality and the identity 
dynamics at play and to actively manage emerging power dynamics 
between herself and the research participants, as well as between the 
NGO and their donors. Emergent themes relating to gender, structural 
inequality and power were dealt with in a relational manner as they arose 
during engagements. As a possible limitation of the study, the authors did 
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not conduct a structural and power analysis or an intersectional analysis, 
as this was beyond the scope of the research. 


E Findings 


To explore what it means, in practice, for evaluators to stay true to MAE’s 
purpose of social justice and equity, we structured our study around three 
main questions: 


e How do evaluators negotiate tensions and power dynamics in the 
evaluation ecosystem? 

e How can evaluators stay true to the lived experiences of programme 
participants? 

e How can evaluators best ensure inclusivity and representation in the 
evaluation processes? 


Firstly, our study revealed that when applying an MAE approach, the role 
of the evaluator extended beyond its traditional scope to include the 
active facilitation of collaboration between key stakeholders. Secondly, 
staying true to the lived experience of programme participants required 
recognising the subjective nature of success and seeing the world through 
an indigenous lens, looking beyond individuals to an impact ecosystem. 
Thirdly, as there is no one-size-fits-all approach, the evaluator needed to 
take time to build relationships and actively involve community members 
in the co-creation of an evaluation framework that best met their values, 
purpose and realities. 


How do we negotiate tensions and power 
dynamics in the evaluation ecosystem? 


Our study showed that effective contextualisation depended on the 
management of tensions between key stakeholders, including the donors, 
the NGO and the programme participants. Each of the stakeholders 
investigated in this evaluation ecosystem operated in a different context, 
was animated by different objectives and was subjected to different power 
dynamics. The main tensions we observed pertained to the paradigms 
adopted by donors in community development, the differences occurring 
between programme intentions and implementation, the lack of mutual 
contextual understanding and the conflicts between demands of objectivity 
and the need for immersion. Managing the various tensions between (and 
within) stakeholders, rather than glossing over them, was critical for the 
creation of an enabling environment for MAE to occur. Key to an MAE 
approach was the active facilitation of collaboration between key 
stakeholders within the evaluation ecosystem. 
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O Moving towards ‘thinking partnerships’ 


Because of the dynamics associated with funding, donors have historically 
held a lot of power in the evaluation ecosystem. To a large extent, they 
determine the manner in which NGOs engage in community development. 
In this study, we found that most donors settled somewhere along a 
continuum between two contrasting archetypes: the ‘mercenary corporate 
donor’ and the ‘thinking partner’ (N17, interviewee, 20 September 2022). 
These two archetypes were conceptualised by the authors, based on 
interview data. 


The ‘mercenary corporate donor’ archetypes took on a conservative 
and conventional approach to development. An NGO employee (N17, 
interviewee, 20 September 2022) described them as ‘shallow, ego-led, 
bottom-line led and scorecard-led’. The donors in this category appeared 
to be motivated by gaining a strategic benefit by funding the programme: 
they believed that giving money would have a positive effect on their image 
and serve as a competitive advantage within the market sphere (D12, 
interviewee, O7 September 2022). These donors followed a donor-led 
approach, meaning they did not engage with community members around 
programme development. Their interventions were described as ‘putting 
band-aids on bullet wounds’ and their impact as ‘palliative at best’ (D12, 
interviewee, O7 September 2022). When it came to measuring the impact 
of these programmes, these donors adopted a quantitative ‘return-on- 
investment’ approach that failed to recognise the lived experiences of the 
participants. 


In contrast, the ‘thinking partner’ archetypes adopted a more flexible 
and participatory approach. According to an NGO employee (N17, 
interviewee, 20 September 2022), these donors believed that ‘charity will 
not solve the ills of the world’, and thus, they preferred to direct their 
contributions through a social investment fund. They were genuinely 
invested in the impact of the programme and were ‘not worried about the 
icing’ (N17, interviewee, 20 September 2022). A thinking partner donor was 
perceived as an active participant in the change process. Rather than 
directing the intervention from a boardroom, they prioritised spending 
time on the ground to immerse themselves in the context and were willing 
to relinquish their power in the system to ensure that communities remained 
the drivers of their own development. Because of their understanding of 
the context, thinking partners were aware of the time necessary for 
practising community development. They were experimental in their 
approach and saw failure as a learning opportunity. In terms of measuring 
impact, thinking partners allowed the NGO to put forward metrics that 
they believed to be indicative of success within the given context. 


While most donors settled somewhere along a continuum between the 
two archetypes, all were acutely aware that the longevity of their business 
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rested on having a more socially equitable and ecologically sustainable 
society. In our study, donors were aware that large amounts of money have 
been invested into programmes that are failing to deliver meaningful 
change. Thus, donors acknowledged that to see a sustainable return on 
their investments, they needed to change the way in which they approached 
development. Despite their varying paradigms, when actively engaging in 
these issues, donors were generally open to adapting their current 
evaluation frameworks to better suit the African context. 


O Acknowledging tensions between methodology 
and practice 


In our study, donors decided which socio-economic challenges were a 
funding priority, and the NGO mostly developed their organisational 
offering to align with those areas. According to an NGO employee: 
‘L..] the South African corporate and public sector currently want to fund 
anything youth focused. If you have an enterprise development spin on it, 
that adds significant icing. And if you add a climate resilience piece on top of 


that, it has a big impact on your chances of being awarded the funding.’ (N17, 
interviewee, 20 September 2022) 


The NGO acknowledged being opportunistic - ‘instead of hitting your head 
against a brick wall where there is no funding, you follow [the trends]’ (N17, 
interviewee, 20 September 2022). In light of this, over the last year, the NGO 
received twice as much funding as in the previous year, with this 100% growth 
being entirely directed to the youth and enterprise development space. 


Following donor funding trends in such a way sometimes caused tension 
between methodology and practice within the NGO. It meant the NGO had 
to deliver on donors’ expectations while doing what they could on the 
ground to honour the community-led process. The tensions came through 
strongly in one of the NGO employees’ (N17, interviewee, 20 September 
2022) descriptions of ‘selling one’s soul to the donor’ and the rest being 
‘smoke and mirrors’. Instead of focusing on the implementation of the 
programme, NGO employees had to spend valuable time and energy 
engaging in what an NGO employee (N14, interviewee, 19 September 2022) 
described as a ‘push and pull’ with donors. 


These tensions required careful management from evaluators, who were 
often required to adapt inappropriate frameworks and evaluation metrics 
to the context and chop and change programme reporting to align with 
the donor’s demands. In doing so, the focus of the intervention was shifted 
away from the participants towards the needs of donors, reinforcing the 
imbalance of power between ‘donors’ and ‘recipients’. In this context, the 
challenge for the evaluator was to find ways that reflected the success of 
the programme in a manner that both met donor demands and honoured 
the lived experiences of the programme participants. 
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O Promoting mutual contextual understanding 


The lack of mutual contextual understanding often led to unrealistic 
expectations. For example, while the entrepreneurship programme was 
pitched to the informal economy, the donors initially thought they were 
funding individuals wearing suits and ties, sitting in an office. However, the 
reality was that the entrepreneurs were often sitting under a tree, selling 
sweets to school children. While this did not make the entrepreneurs’ 
business venture any less valid, it did impact the kind of challenges they 
faced and the type of support the programme needed to provide to ensure 
success. As mentioned by one of the donors (D13, interviewee, O7 September 
2022), ‘you can’t copy-paste an academic model into a rural area and expect 
it to work’. It was not until the NGO organised for the donors to visit the 
region and meet the entrepreneurs that the donors realised the reality of 
the programme and acknowledged the need to adapt it to the context. 


This method of bringing one stakeholder into the physical space of 
another, to improve their understanding of the other’s context, has been 
evidenced as a successful way to manage stakeholder tension. For instance, 
bringing donors into the physical space has been a successful method used 
by the NGO to assist the donor in understanding the contextual factors that 
might be impacting the programme’s success. One of the donors (D13, 
interviewee, O7 September 2022) said that site visits were a ‘must have’ in 
the reporting cycle, as written reports ‘did not do justice to reality’. 
Depending on the type of donor, the NGO sometimes ‘sanitised [the visit] 
for them’ (N17, interviewee, 20 September 2022). However, other donors 
were taken ‘straight to the laundry room’ (N17, interviewee, 20 September 
2022). In some cases, it was not possible for the donor to conduct a site 
visit, and thus, there were times when the NGO organised for the 
entrepreneurs to visit the donor in another region, for example, Johannesburg. 
This had an additional benefit as the entrepreneurs were then able to better 
understand the context in which the NGO and donor operate. 


The need to align expectations and contextual understanding also 
proved important for improving internal dynamics within organisations. 
For instance, if the NGO’s head office team was not exposed to the regional 
work context (and vice versa), tensions would arise. The NGO employees 
thus organised site visits for the head office team (based in Johannesburg) 
to visit the regional team, and vice versa. An NGO employee (N15, 
interviewee, 20 September 2022) shared how, since her visit to the 
KwaZulu-Natal region, she switched from making online calls to using 
WhatsApp to communicate with her colleagues, as she had personally 
experienced how bad the network was. She also mentioned that she is now 
more understanding when data sheets are submitted late as she has seen 
how some of the rural communities are set in mountainous areas where 
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travelling is slow. The same employee also spoke about the benefit of the 
regional team visiting the head office, as they got a better idea of the 
pressure and demands that the head office faced. 


O Fostering buy-in on the value of immersion 


Donors expected evaluators to remain objective and separated from the 
people and programmes they were evaluating. However, an MAE approach 
called for evaluations to be conducted with participants and not on them 
(Chilisa 2019). This required that evaluators build relationships with 
programme participants and immerse themselves in the context. In this 
study, the evaluator explained the difficulty she faced in reconciling these 
conflicting demands. She explained that ‘the idea that we need to be 
objective, or that it is even possible to be objective, is completely outdated’. 
Through her experience evaluating these programmes, the evaluator further 
found that ‘evaluations are about experiences, judgements, mistakes, new 
ideas, successes, and multiple voices - we should not cut these out!’ 


As the MAE approach required the evaluator to embed themselves in 
the context - embracing and reflecting on the subjective notions of impact 
that emerged - the donors’ expectation of objectivity needed to be 
managed carefully. Rather than working against the donor, the evaluator 
made conscious efforts to engage with the donors, explaining why 
qualitative metrics mattered, what insights they would gain from 
understanding the lived experiences of the programme participants and 
what they could expect from the evaluation report. Donors were receptive 
to the changes - one donor (D11, interviewee, 31 August 2022) responded, 
‘if it improves the work we're doing, then why not!’. 


How do we stay true to the lived experiences of 

participants? 
Before the study, the NGO was largely making use of a quantitative approach 
to measuring success. This approach did not honour the lived experiences of 
the programme participants, and, in response, the evaluator spent time 
engaging with programme participants to better understand their individual 
perspectives of success. Through these engagements, we found that success 
is subjective and largely experienced in relation to others. 


CO Recognising the limits of a ‘return-on-investment’ 
approach 


Both programmes made use of a similar quantitative evaluation framework, 
which could be described as focused on ‘return on investment’. Assumptions 
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had been made about programme recipients and their needs, and 
evaluations were conducted on them, using predominantly quantitative 
metrics to measure inputs and outputs. For example, in the work experience 
programme, the NGO measured how many participants were onboarded, 
how many dropped out and how many were subsequently employed. 
This approach to measuring success focused on the breadth of impact 
(X number of participants reached) rather than the depth of impact 
(participants experienced X change). 


While this approach was aimed at objectivising the measurement of 
impact, it failed to consider the humans behind the data. It obscured the 
motivations driving people to engage in the programmes, and it did not 
give programme participants the opportunity to express their understanding 
or views of success. For example, a programme participant (P2, interviewee, 
O7 July 2022) expressed that her motivation for joining the entrepreneurship 
programme was to ‘broaden her interaction with people in the community’. 
Another programme participant mentioned that she ‘struggled with mental 
health and substance abuse’ and she joined the work experience programme 
so that she could ‘have a reason to get up in the morning’ (F10, interviewee, 
26 August 2022). Thus, while the programme might have been developed 
to grow an individual’s business or assist them in getting a job, each 
individual was motivated by personally specific desires and goals. 


CO Capturing the impact ecosystem 


Programme participants expressed their experience of success mostly in 
relational terms, which can be hard to quantify. For instance, some 
participants experienced success or failure depending on how well they 
felt heard and supported by programme leaders. Others mentioned that 
their experience of success was related to the fact that their family and 
community treated them with greater respect than before. These 
experiences, along with others, such as gaining independence, learning 
various soft skills and feeling a greater connection to nature, all represented 
experiences of success that do not currently feature in the donor reports. 


While some participants focused on how the programme impacted 
them on an individual level, most people relayed their experience of success 
in relation to others, such as the impact that their participation in the 
programme had on their families, the community and the natural 
environment. For example, participants in the focus groups (F9 and F10) 
spoke about success as being able to afford school uniforms for their 
children, contribute to groceries and other household expenses and share 
new skills with their family members. On a community level, participants 
saw success as providing goods and services locally so that community 
members did not have to spend time and money travelling to larger towns. 
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On an environmental level, participants highlighted positive behaviour 
shifts (e.g. away from littering), the preservation of indigenous medicinal 
knowledge systems and an overall change in attitude towards protecting 
wildlife as the underlying successes of the programme. In sum, an impact 
ecosystem looks at how impact is experienced beyond the individual to 
include the relationships within and between the surrounding community 
and environment. This framing of success, in terms of an impact ecosystem, 
better captures the relational paradigm that guides Zulu culture. 


O Acknowledging the subjectivity of success 


Unpacking how participants experienced success in relation to others 
helped to contextualise impact. For instance, the varying experiences of 
success shared by participants revealed the influence of culturally defined 
gender roles in this context. In the case of the entrepreneurship 
programme, we found that, in general, a successful day for a female 
entrepreneur was one in which she could buy groceries and care for her 
family, while male entrepreneurs generally saw success as being able 
to build a house for their family or grow their assets. Thus, if the 
entrepreneurship programme was to measure success in terms of profit 
only, it would fail to recognise the gendered experiences of individual 
success that are present in this context. 


While NGO employees appeared to understand the importance of 
capturing the subjective and relational dimensions of success using more 
qualitative measures, they were reluctant to use qualitative metrics because 
these were not as easy to define and measure as quantitative metrics. As 
such, one NGO employee (N15, interviewee, 20 September 2022) recognised 
that ‘success is different depending on whose perspective you are looking 
from’. Even the donors in our study recognised the need to explore and 
implement more qualitative data-collection methods. As one donor (D1, 
interviewee, n.d.) said, ‘we need to ask more intentional and direct questions 
about the experiences of the programme participants’. However, qualitative 
measures are difficult to design and take time to implement. As NGO 
one employee (N15) highlighted, ‘the challenge of understanding what 
those metrics are, and how to capture them, means they get blanked out 
quite quickly’. 


O Understanding how the context impacts success 


Thus, part of the evaluator’s work entailed understanding the contextual 
factors that were affecting how the participants engaged in a programme 
and how they experienced success. An NGO employee (N15) provided a 
good example of how the value of respect, which is highly regarded in Zulu 
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culture, can impact the manner in which a participant communicates within 

the programme: 
‘A lot of participants come from family structures where there is dominance or 
hierarchy and so they tend to see a boss as someone they can’t open up to. It 
also goes back to the kind of education that we [Zu/us] had growing up. We were 
not allowed to challenge or question older people. So rather, the participants sit 
with questions. And at the end of the day, they don’t learn because they don’t 
ask questions.’ (N15, interviewee, 20 September 2022) 


This type of information helped to explain why participants may not have 
experienced the programme as successful or useful to them. It provided 
nuance to the evaluation and was useful information for the design of 
future interventions in the community. Using this example, future 
interventions might ensure that programme participants either have 
someone who they perceive to be an equal to talk to or that they are made 
to feel comfortable speaking to someone despite their perceived seniority. 


Another contextual factor that was found to affect how participants 
engage in the programmes was the patriarchal nature of the Zulu culture. 
This cultural belief continues to shape what females are or are not allowed 
to do. One programme participant (P2, interviewee, n.d.) shared, ‘there is a 
culture among us Zulu people that females are not supposed to work - 
their responsibility is to be a housewife and take care of the kids’. Despite 
this, females are increasingly becoming breadwinners in their families as 
males migrate to cities to find stable work. However, there are still cultural, 
gendered restrictions on acceptable work. According to one programme 
participant (P7, interviewee, n.d.), a male would find it shameful to sell fruit 
and vegetables and a female would never be accepted as a taxi driver. 
According to another programme participant (P1, interviewee, n.d.), females 
are also restricted in what they are allowed to wear and are not meant to 
be outspoken. These cultural elements had very real implications for how 
participants, across genders, engaged within the programmes. Thus, a 
culturally responsive evaluation would need to navigate how gender norms 
and roles impact a participant’s experience of success on a programme. 


How do we ensure inclusivity and representation 
in evaluations? 


Despite the NGO already having an established evaluation framework, 
part of this study involved facilitating focus groups (F9 and F10) with the 
programme participants to see how they related to these methods and 
metrics and what could be done differently. Our study revealed that 
creating a contextually relevant evaluation framework meant addressing 
identity dynamics and working on an ad-hoc basis, as the usefulness of 
the evaluation framework depended on the active participation of 
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programme participants. In our context, programme participants had a 
clear idea of why evaluations were important, what metrics should be 
measured and how the data should be collected. Importantly, the process 
of designing the evaluation framework offered all involved the time and 
space to build relationships, build trust and for co-creation to take place. 


O Addressing identity dynamics 


In our study, the evaluator was an ‘outsider’ to the context, and this carried 
several implications for her practice of MAE. Firstly, the evaluator 
recognised the limitations that her identity presented and acknowledged 
that she may never be able to fully honour an ‘indigenised approach’ 
(Chilisa 2019) to evaluation as her lived experiences have not equipped 
her with a deep enough understanding of indigenous African culture. She 
responded to this limitation by adopting an ‘adaptive approach’ (Chilisa 
2019) while all the time remaining open to learning more about Zulu 
culture and the traditional belief systems, values and practices present in 
the context. As time went by, she noted that her developing knowledge 
of the context strengthened her practice of MAE. 


Secondly, the evaluator experienced how her identity influenced the 
types of responses she received from programme participants. She found 
that many participants answered with what they thought she wanted to 
hear rather than with how they felt. When this dynamic was interrogated, 
we found it to be linked to two contextual factors. On the one hand, the 
participants felt it necessary to show respect to the evaluator, and it would 
not have been considered respectful to say anything bad about the 
programme. On the other hand, the participants were deeply reliant on the 
programmes and thus did not want to say anything to compromise future 
interventions. This tension was managed by involving local NGO employees 
in interviews and focus groups and reassuring the participants that there 
was no risk of the programme being halted - their responses would only 
help to strengthen the programme. 


Thirdly, the language barrier between the evaluator and the participants 
posed an additional challenge, as it limited the extent to which the evaluator 
could present a narrative that truly reflected the lived experiences of the 
programme participants. When selecting programme participants to 
interview, the evaluator had to make a conscious effort not to exclude 
certain voices based on their English language proficiency. She overcame 
this challenge by sharing the responsibility of identifying interview 
candidates with her local research assistant. The presence of a local 
research assistant meant that all interviews could be conducted in either 
English or isiZulu, depending on the preference of the participant, helping 
to ease the communication barrier. 
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O Co-creating evaluation frameworks through 
active participation 


We found that to create a contextually relevant evaluation framework that 
speaks to the lived experiences of the community, programme participants 
must actively participate in the evaluation design process. They need to 
participate in defining how they want data to be collected, on what aspects 
of the programme and for what purpose. The evaluation design needed to 
consider the fact that different data were likely to be of different value to 
different stakeholders. Discussing the implication of our findings, one of 
the donors recognised that, in the future, ‘evaluations will be defined on an 
ad-hoc basis, by the communities in which the funding is being dispensed’ 
(D12, interviewee, O7 September 2022). 


An NGO employee (N18, interviewee, n.d.) made the important distinction 
between ‘involvement’ and ‘active participation’ when it comes to including 
the community in design processes. He described involvement as a ‘by the 
way’ thing - community members can sit in a meeting and ‘be involved’ yet 
still have no agency in the process. Active participation, however, requires 
you to have ‘skin in the game’ (N18, interviewee, n.d.). Through active 
participation, ideas are shared, robust discussions take place and community 
members are given the platform to define their own measures of success. 


Part of the aim of the focus groups was to engage with programme 
participants about how data could be collected in a contextually appropriate 
manner. We found that the preferred methods, as presented by the participants, 
differed across the programmes. Participants from the entrepreneurship 
programme mostly chose verbal methods of data-collection such as 
storytelling, video clips and presentations, and they chose to produce these 
in their home language, isiZulu. Participants from the work experience 
programme chose mostly written methods of data-collection, such as surveys, 
pictures with captions and WhatsApp, and they chose to produce their 
responses in English. These responses were in line with the varying education 
levels of the participants on each programme: participants on the 
entrepreneurship programme generally had not completed high school, and 
thus, their language and writing skills were not as well-developed, whereas 
many participants on the work experience programme had completed tertiary 
education. Thus, in this context, it was found that an appropriate evaluation 
framework would require an adapted data-collection strategy based on the 
varying educational levels of participants across programmes. 


O Embracing the plurality of values and ways of 
engagement 


Whereas evaluations are traditionally designed to be conducted for the 
benefit of the donor, so that they can track the impact of their spending, 
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we found that it was as important to ensure that the collected data were 
relevant and useful to the programme participants themselves. When asked 
in the focus group ‘why’ we collect data, participants mentioned the 
usefulness for donors, but they also explained that the data were useful to 
them. One of the entrepreneurs in the focus group (F9, interviewee, n.d.) 
said that the financial data that they were required to collect were ‘helping 
her to better manage her business’. Another participant from the focus 
group (F10, interviewee, n.d.) said that they wanted their data to be linked 
to their name (rather than anonymous) as they could then market 
themselves through the data-collection process. Further, we found that 
those who participated in co-creating the evaluation framework made 
greater efforts to provide the correct data in a timely manner. 


During the focus groups (F9 and F10), indigenous methods of data- 
collection were presented and discussed with programme participants to 
assess how they could be integrated into the evaluation framework. As an 
example, the evaluator described how the indigenous practice of storytelling 
could be used in evaluation. Despite opening the space for creative 
engagement on the topic, programme participants showed little interest in 
exploring these methods. However, what did come through was the notion 
that young Zulu people ‘live in two worlds - one Western, one traditional’ 
(F9, interviewee, n.d.). This meant that they were comfortable using 
platforms such as WhatsApp for data-collection and participants said that 
methods such as this were ‘fast, simple, convenient and cost-effective’ 
(F10, interviewee, n.d.). It was thus found that a contextualised evaluation 
framework does not necessarily need to be focused on indigenous methods. 
Rather, it needs to be sensitive to the context of the target group and offer 
relevant options. It is important to emphasise that the focus group was 
made up of youth aged 18-35-years-old. Should the focus group have 
contained older Zulu people from the community, the level of interest 
towards using indigenous methods might have been different. 


O Allowing time and space to build trust and maintain 
relationships 


The focus group sessions (F9 and F10) that were held to design the 
evaluation framework showed the programme participants that their 
input was valued and assisted in building trust between themselves, the 
evaluator and the NGO employees who were involved in the process. Yet, 
this process took time and was better conceived as an ongoing process. 
As an NGO employee (N15, interviewee, 20 September 2022) observed, 
‘you can’t push things and you can’t be pressured by time - you must follow 
the correct processes’. Another NGO employee (N19, interviewee, 10 May 
2022) further added, ‘good stakeholder relationships are critical to 
community development processes and most people don’t realise how 
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much work goes into maintaining these relationships’. Thus, we found that 
these evaluation design sessions should not be considered as a once-off 
process but could rather be designed as iterative efforts to reflect and 
improve on the evaluation design in collaboration with the programme 
participants. 


One of the key benefits of taking time to build relationships within the 
evaluation ecosystem was allowing the evaluator to identify local NGO 
employees who would be capable of taking ownership of the evaluation 
design and implementation in the future. Over time, the evaluator involved 
these NGO employees in preparing for the focus groups and helped them 
develop the skills to facilitate the focus groups themselves. This helped to 
ensure that the focus groups were responsive to the context, assisted in 
neutralising insider-outsider power dynamics between the evaluator and 
the programme participants and removed the language barrier. As pointed 
out by an NGO employee (N15, interviewee, 20 September 2022), ‘when 
participants see someone from another race, their minds go straight to 
funding; but when they see someone from the community, they are much 
more open’. It was seen that with further training, these local evaluators 
would become well-positioned to discover new ways to evaluate 
programmes that align with the desires and the needs of community 
members and that better reflect the lived experiences of the programme 
participants. 


E Conclusion 


Contributing to the growing body of evaluation research, our study provides 
insights into the role of the evaluator in operationalising an MAE approach 
in a way that is truly rooted in the context and promotes social justice and 
equity. Firstly, we found that operationalising an MAE approach requires 
the evaluator to acknowledge and then actively manage any tensions and 
power dynamics present in the evaluation ecosystem. There is an 
opportunity for corporate donors to reposition themselves as ‘thinking 
partners’, to better align their approaches with the needs and interests of 
the programme participants. Part of this process involves bringing 
stakeholders into the physical space to promote a contextual understanding. 


Secondly, while MAE scholars (Chilisa 2019; Khumalo 2022) are pushing 
for the decolonisation of evaluative practice and encouraging a move 
against Euro-American evaluation frameworks, there is a risk that the 
language of MAE scholars is creating an ‘us and them’ divide between 
practitioners and donors. Our study shows that corporate donors are 
acutely aware that their investments are not effectively translating into 
impact, and if carefully guided and supported, they are often willing to 
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adapt their frameworks to better suit the African context. To successfully 
integrate an MAE approach, evaluators need to work with donors, not 
against them. 


Thirdly, to stay true to the lived experiences of participants and ensure 
that their values matter in evaluations, evaluators need to consider how 
success is experienced from an indigenous viewpoint. In our research, this 
meant looking beyond individual impact and considering the entire impact 
ecosystem. Metrics to measure success should be developed on an ad-hoc 
basis and be rooted in the context to accurately capture the participants’ 
experiences. 


Fourthly, to ensure inclusivity and representation in evaluations (Khumalo 
2022), evaluators need to take the necessary time to foster relationships 
of trust among stakeholders. Further, community members need to be 
actively involved in the co-creation of an evaluation framework that suits 
their needs and priorities. While pursuing Chilisa’s (2019) indigenous 
agenda, we also need to recognise that the world is changing, and in some 
cases, the context might call for a mix of both Western and African cultural 
elements in evaluative practice. 


Finally, the insights from this study contribute to MAE practice by providing 
evaluators with guidance on how to implement Africa-based evaluations in a 
way that promotes social justice and equity while remaining sensitive to the 
needs and expectations of all stakeholders. In doing so, this study helps guide 
evaluators on how to hold space to negotiate tensions and power dynamics 
in the evaluation ecosystem, embrace indigenous worldviews, stay true to the 
lived experiences of programme participants and ensure inclusivity and 
representation in evaluation. 
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Programme evaluation as a 
reflection and perpetuator 
of inequality in the global 
political economy 


Ayabulela Dlakavu 

Twende Mbele, 

Faculty of Law, Commerce and Management, University of the Witwatersrand, 
Johannesburg, South Africa 


E introduction 


The programme evaluation discipline and practice have historically been 
driven by Global North intellectuals and institutions, the latter inclusive of 
bilateral donors (member countries of the Organization for Economic 
Co-operation and Development [OECD]) and multilateral donors (such as 
the United Nations [UN] and World Bank). The political, economic and 
ideological hegemony of the Global North in the international system has 
translated into a monopoly on the conceptualisation and measurement of 
international development, determining which development indicators 
should be measured, how and why. Hegemony historically reinforces itself 
through enduring ideology and institutions. It is no surprise, then, that the 
founding and evolution of evaluation theory and practice have been 
sustained by OECD donors and intellectuals, UN agencies and Bretton 
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Woods Institutions (i.e. the World Bank). The Global North’s monopoly of 
evaluation discourse and practice (including determining key evaluation 
criteria and framing of evaluation capacity development [ECD]) is a 
mechanism through which this historically hegemonic geopolitical region 
seeks to maintain its strong influence on Global South development 
discourse, measurement and governance. 


As a result, an examination of evaluation theory and practice prevalent 
in Africa will reveal that Global North institutions, such as the World Bank 
and UN agencies (largely funded by OECD countries), are the primary 
drivers of evaluation approaches and ECD initiatives. This chapter argues 
that evaluation approaches and practice are dominated by Global North 
neoliberal conceptualisations of what constitutes deve/ooment and how to 
measure it. Such Western monopolisation of evaluation discourse and 
practice is areinforcement of the Global North’s monopolisation of ideology 
and knowledge in the global political economy, owing to the post-1945 
neoliberal institutions that are able to advocate for the adoption of 
neoliberal ideology and policy worldwide. The dominance of Global North 
development and evaluation thinking deprives these two practices of their 
transformative and emancipatory potential in Africa. This chapter provides 
a critique of current development and evaluation discourse and practice in 
Africa. From the critique, the chapter then moves to proffer recommendations 
on how to bring about development and evaluation discourses and 
practices that articulate African development needs and aspirations, and 
development measurement mechanisms that are able to capture real 
development experiences in Africa and offer pathways to socio-economic 
development. When African development and evaluation discourse is 
preoccupied with matters of equity and inspired by African perspectives 
of what constitutes development rather than Global North quantitative 
measures, the continent should then be in a position to pursue real socio- 
economic development that makes the continent developmentally 
competitive and comparative with other geopolitical regions with advancing 
economies. The added value is that such an Afrocentric development 
discourse and practice will elevate African development theory and policies 
and African development measurement approaches (i.e. evaluation 
approaches). This is one way of advancing African intellectual thought in 
development and evaluation discourse and practice within the global 
political economy, thereby advancing intellectual equality. 


E Methodological approach 


This chapter adopts a qualitative design. A review of performance 
monitoring and evaluation (M&E) literature, as well as reflections on 
M&E practice (i.e. action research), literature on conceptualisation of 
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development, and the author’s observation as a development and 
evaluation practitioner are the complementary methodologies adopted. 
Action research is the overarching qualitative research method discussed 
in this chapter. Popplewell and Hayman (2012, p. 1) define action research 
as research undertaken by any practitioner with the aim of identifying 
problems in a particular practice or discipline and consequently proffering 
remedial solutions to such problems. Action research seeks to improve 
the methods and approaches of a particular discipline or practice. In the 
context of this chapter, the author reflects on dominant conceptualisations 
and indicators of development in Africa, as well as reflecting on African 
evaluation approaches and practices as codified in the literature. Both 
these inseparable practices (development and evaluation) are interrogated 
to examine the extent to which they prioritise equity and inequality. 


Direct observation is another complementary qualitative research 
method adopted by this chapter. Ciesielska, Bostrom and Ohlander (2008, 
p. 34) define observation as a researcher’s reflections on the successes 
and challenges of a culture, a people or practice, and using this reflection 
to propose recommendations on improving the observed subject. In 
research, observation is undertaken in a deliberate and systematic manner 
whereby the researcher watches, evaluates and draws logical conclusions 
on the observed subject. In this chapter, the author’s analysis of 
development thinking, practice and evaluation is also informed by his 
experiences and observations as a student, scholar and practitioner in the 
fields of development and programme evaluation. Recommendations on 
how to dismantle the Global North’s monopoly of development and 
M&E discourse and practice are provided in the closing segment of the 
chapter, which recommends transformative and equity-based programme 
evaluation in Africa. 


Mohajan (2018, p. 15) defines content analysis as a detailed and 
systematic examination of a body of work with the intention of identifying 
patterns, themes or biases. In this chapter, the author examines the literature 
concerning development and programme evaluation in order to understand 
the core meaning of these concepts and how they are applied in Africa. 
Thus, the sections of the conceptual framework of the chapter and the 
overview of adopted evaluation approaches and practices in Africa are 
based on content analysis. 


Action research, content analysis and direct observation are therefore 
the complementary qualitative research methods employed by this chapter 
to prove or disprove its hypothesis that develooment and programme 
evaluation discourses and practices are a mirror and perpetuator of 
inequality in the global political economy, and one of the manifestations of 
this inequality is the dominance of Global North ideology and institutions 
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in shaping development knowledge and setting development indicators. 
The author concludes the chapter by making practical recommendations 
for the positioning of equity as a central focus of African development and 
evaluative practices. 


E Conceptual framework 


The aim of this chapter is to scrutinise the hegemony of the Global North 
in programme evaluation discourse and practice, and it is therefore 
important to begin with an interrogation of the core concepts of this 
ensuing body of work. The key terminology that constitutes the chapter’s 
conceptual framework includes development, equity, programme evaluation, 
inequality and global political economy. These are central concepts when 
interrogating current programme evaluation discourse and practice and 
formulating normative recommendations for an equity and context- 
relevant programme evaluation discourse and practice in Africa. 


Conceptualising programme evaluation 


Programme evaluation is defined as a learning tool that describes and 
explains the results (outcomes and impact) achieved by a development 
intervention (i.e. a policy or programme) vis-a-vis its stated objectives 
(Rallis & Bolland 2005, p. 5). Programme evaluation findings can be used 
to improve the implementation of a given programme (e.g. investment of 
resources and roll-out of activities) or to make decisions about the future 
of a development policy or programme (i.e. scaling up, continuing or 
discontinuing the intervention). The ultimate aim of programme evaluation 
is to help a development policy or programme to achieve the development 
outcomes it was intended to deliver (Rallis & Bolland 2005, p. 6). Programme 
evaluation is therefore an agent or tool of social change or transformation 
in development practice. 


For its part, the Dutch Ministry of Foreign Affairs (2009, p. 15) defines 
evaluation as a systematic and objective assessment of the design or 
implementation of an ongoing development policy, programme or project, 
or alternatively, the results achieved by a policy, programme or project. In 
essence, an evaluation seeks to measure the relevance, coherence, 
efficiency, effectiveness, impact and sustainability of a development 
intervention. The ultimate purpose of an evaluation is to unearth findings 
or evidence to inform decision-making about the policy, programme or 
project. From the above definitions, the common discernible message is 
that evaluation serves as a performance management and learning tool for 
policy-makers, programme and project managers. Its core aim is to ensure 
optimal policy, programme or project performance and to induce better 
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development outcomes. Given the intertwined nature of eva/uation and 
development, the latter concept is explained further in the text. 


Conceptualising development 


The widely accepted definition of development is coined by the Global 
North nations and developmental institutions. The Global North considers 
development as being characterised by the growth of a country’s gross 
national product (earnings from all goods and services produced and sold 
by a country, and the country’s earnings from foreign investments), a 
growth in per capita incomes, steady industrialisation and technological 
advancement (Abuiyada 2018, p. 115). These are the indicators of 
development from a Global North perspective. Contrary to this enduring 
conceptualisation, Indian economist Amartya Sen defined development as 
a process of eradicating poverty, tyranny, lack of economic opportunities 
and eradication of social deprivations (Tavernaro-Haidarian 2019, p. 21). 
Sen, as a Global South economist, therefore understands development 
from an equity perspective rather than the limited neoliberal and 
quantitative indicators of development found in Global North 
conceptualisations of development. The dominance of neoliberal 
perspectives on development has been sustained by the Bretton Woods 
Institutions (World Bank and International Monetary Fund) and other 
Global North developmental institutions. 


Conceptualisation of development and its indicators are important 
because they inform public policy and foreign policy choices of governments 
and also inform development aid (development co-operation) strategies of 
Global North states vis-a-vis the Global South. Moreover, development 
definitions and its indicators are important because they are a primary unit 
of measurement in evaluation. If definitions and characterisations of 
development are wrong, then evaluations will measure the wrong things 
when determining the effectiveness or impact of public policies in the 
quest for development. 


Over time, governments and non-governmental development 
practitioners have realised that development interventions informed by 
Global North conceptualisations of development have not been sufficient 
in addressing poverty and access to opportunities for empowerment and 
upward social mobility for the majority of Global South populations. In fact, 
one of the major issues with development in the Global South is that it has 
served the material interests of select groups at the expense of others, a 
condition known as inequality. Inequality is defined as the unequal 
distribution of wealth, resources and opportunities that enable individuals 
to live a good life free of need. Development inequality within a country is 
manifested by unequal distribution of income, wealth, expenditure and 
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opportunities between individuals and groups on the basis of gender, 
ethnicity, geographic location or family background (Dabla-Norris et al. 
2015, p. 6). The author augments this definition of inequality by arguing 
that intrastate inequality is a structural condition that has its roots in 
historically unjust legislation that advanced the interest of one group in 
society at the expense of others (on the basis of race, ethnicity or gender). 
This structural inequality has been perpetuated by development 
interventions that do not address these historical structural causes of 
inequality. 


The author makes the following observations about development 
inequality in the global political economy (i.e. unequal distribution of 
economic and political power among nations). Firstly, one observes that 
development inequality in the global political economy exists among 
nations as a result of unjust historical processes such as the post-1500 
European age of ‘discovery’ of foreign lands (already inhabited by 
indigenous peoples), the trans-Atlantic Slave Trade, colonialism and neo- 
colonialism following the granting of political independence to former 
colonies. From the author’s observations, the historically marginalised 
geopolitical regions such as Africa, Asia and Latin America are compelled 
to achieve socio-economic progress within a global political economy 
designed to serve the economic interests of the Global North through 
instruments such as foreign direct investment (FDI), the international free 
trade regime implemented by the General Agreement on Tariffs and Trade 
(GATT) and its successor, the World Trade Organization (WTO) (which 
remove the capacity of Global South governments to protect domestic 
producers from international competition) and the development loan 
conditions of international financial institutions C(IFls) such as the World 
Bank and the IMF. 


The FDI grants foreign investors a disproportionate influence over the 
economic trajectory of a nation’s economy, especially in the context of the 
financialisation of the global economy since the 1970s, whereby many 
national governments have removed capital controls, allowing money to 
flow in and out of an economy without much government regulation. The 
risk for developing economies is that they have been subjected to financial 
shocks when unregulated capital suddenly leaves a domestic economy, 
leaving many businesses collapsing, unemployment skyrocketing and 
poverty rising. Examples of this are the Asian Financial Crisis of 1997 and 
the 2008 Global Financial Crisis, two financial sector and FDlI-induced 
crises that led to the collapse of national economies, felt most acutely by 
the Global South countries. Historically, IFls have also adopted development 
loan conditions that have led to economic underdevelopment in the Global 
South. The author gives the example of the neoliberal Structural Adjustment 
Programs (SAPs) that were imposed on borrowing Global South countries 


52 


Chapter 3 


in the 1980s and 1990s, which compelled Global South governments to cut 
expenditure on state subsidisation of food, education and health care, 
which led to adverse socio-economic effects such as rising food prices, 
unemployment and poverty in Africa and Latin America. 


Jones (2009, p. 3) defines equity as a normative conceptualisation of 
development that seeks to achieve equality of opportunity, fairness and 
social justice in society. Equity is concerned with three principles: (1) equal 
life chances and socio-economic opportunities for all people in society 
regardless of race, class and gender; (2) equal concern for people’s needs 
and distributing goods and services without discrimination; and (3) 
meritocracy, where positions, opportunities and rewards are based on 
fairness, effort and ability (Jones 2009, pp. 6-7). In the context of this 
paper’s focus on whose knowledge matters when conceptualising 
development and measuring it (evaluation), inequality is a key concept that 
requires framing. The UN Department for Economic and Social Affairs 
(2015, p. 1) defines inequality as a condition whereby individuals (or nations) 
are not equal regarding status, rights and opportunities. From this framing 
of equity and inequality, one can observe that equity seeks to alleviate the 
phenomenon of inequality. This chapter highlights the global political 
economy’s facilitation of knowledge generation inequality with the aim of 
advocating for a normative future where the Global South and its institutions 
can also shape ideas about development and how to drive and measure it. 


One of the notable failures of development and globalisation is that 
people’s access to opportunities and resources are determined by their access 
to economic and political power. Individuals who do not have access to 
decision-makers in political and economic institutions often suffer from 
poverty, relative deprivation and social exclusion, and these unfortunate 
outcomes are quite acute in the Global South. The inequality, socio-economic 
and political exclusion then give rise to developing countries’ domestic 
instability in the form of high crime rates, uprisings and revolts. In order to 
achieve equity, national and international development interventions (policies 
and programmes) should seek to provide universal public services; government 
expenditure should favour disadvantaged social groups, provision of social 
protection (i.e. basic income grants) to marginalised groups, roll-out of 
redistribution policies (i.e. nuanced tax policies and land reform), and 
addressing power imbalances (Jones 2009, p. vii). In order for programme 
evaluation to be a transformative discipline, discourse and practice, it needs to 
adopt an equity lens when measuring the relevance, effectiveness, impact and 
sustainability of develooment interventions. This means that evaluation criteria 
should be equity-focused and be able to measure the extent to which 
development interventions advance universal access to basic services and 
access to decent employment for intended beneficiaries. In this way, evaluation 
would be able to advance a developmental agenda of ensuring that everyone 
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in society receives equal economic opportunities while affirming their social 
and political rights. 


Global political economy and the hegemony of 
Global North knowledge and ideology therein 


The global political economy refers to the global interactions and integration 
of nation-states and non-state actors through international trade (and the 
WTO governing international trade), international finance (and key IFls 
providing and governing international finance, chiefly the neoliberal Bretton 
Woods Institutions [World Bank and IMF]) and knowledge creation and 
transfer (led by Global North institutions which maintain a stronghold on 
what is considered knowledge and what is not). A key characteristic of the 
global political economy is the dominance of the Global North’s influence 
on international trade, finance and knowledge production and transmission 
(Thompson 2000, p. 4), because of the dominance of neoliberal ideology 
and policy on trade (i.e. economic liberalism and free trade ideas) and 
finance (i.e. the dominance of neoliberal conditions when Global South 
countries apply for finance from Bretton Woods Institutions and other 
institutions of the Global North). In the context of this chapter, the author 
observes that the Global North enjoys a monopoly on knowledge systems 
(knowledge production and sharing, particularly on what constitutes 
development, how to facilitate development and how to measure [monitor 
or evaluate] development, including key evaluation criteria when measuring 
development). The next section of this chapter demonstrates the Global 
North-centric nature of programme evaluation as a discipline (area of 
study), discourse and practice. 


E Overview of evaluation discourse, 
discipline and practice in Africa 


This section explains the evolution of programme evaluation globally and 
in Africa, with the aim of interrogating the discipline’s responsiveness 
to African development realities. Evaluation capacity development 
interventions on the continent are also examined to determine the extent 
to which they are making an effort to transform African evaluation discourse 
and practice into a discipline and practice that makes a meaningful 
contribution to the alleviation of development challenges in Africa. 


Origins and evolution of programme evaluation 


Worthen (1990, p. 42) traces the origins of modern-day programme 
evaluation to the education sector in the United States of America (USA). 
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From 1897 to 1898, there was a comparative evaluation study of the spelling 
performance of 33,000 North American students. In subsequent decades 
of the 20th century, other North American education programmes were 
also evaluated and formed the basis of curriculum reforms. A major flaw 
with these education system evaluations is that they made no reference 
to the objectives of the education programmes being evaluated (i.e. the 
evaluand) and there were no clear evaluation questions to which the 
evaluation reports were responding, which then rendered the data 
somewhat questionable (Worthen 1990, pp. 42-43). Furthermore, 
evaluations in the education sector tended to be quantitative in nature (i.e. 
x amount of students can spell very well while y amount of students cannot 
spell). In the 1960s and 1970s, programme evaluations in the education 
sector transformed as evaluators and researchers began to diversify 
towards qualitative evaluation designs, thus breaking the monopoly of 
quantitative research designs that did not address the why and how 
questions during the course of undertaking an evaluation (Worthen 1990, 
p. 43). With such advancements in evaluation methodological approaches, 
programme evaluation was then adopted and adapted by other development 
and public policy practitioners in other sectors in the late 20th century. 
From this section, it is clear that the origins and evolution of programme 
evaluation are traced back to 19th century USA in the Global North. 


Worthen (1990, p. 47) asserts that a major challenge for evaluators in 
the 1980s was that they failed to understand the political nature of 
evaluation, resulting in irrelevant evaluation reports that lacked value in the 
policy cycle and failed to take note of political forces and factors within the 
public policy cycle. One can argue that the same issue raised by Worthen 
in 1990 persists in the 2020s. Within development sectors and agencies, 
programme evaluation is inherently a political activity because it is meant 
to provide advice to programme managers and policy-makers regarding 
the performance, worth and future of the policy or programme being 
evaluated. Because of being a tool and function of the public policy cycle 
(which is headed by political principals), programme evaluation is therefore 
a political activity within a political process. For example, Eckhard and 
Jankauskas (2020, p. 668) assert that in the public sector, programme 
evaluation is a tool used by political and administration decision-makers to 
help inform policy decisions and the allocation of public resources. Taken 
from this perspective, evaluation is a key input into public sector policy 
decision-making and budgeting, two equally critical instruments that affect 
the trajectory of the socio-economic development of a nation. The political 
nature of evaluations is not limited to state institutions, with Eckhard and 
Jankauskas (2020, pp. 668-669) asserting that the evaluation function in 
international organisations (lOs) is also a political activity because 
evaluation budgets are determined by the resources committed by member 
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states to IO evaluation departments. Furthermore, member states of IOs 
often use evaluation reports by IO departments to influence and determine 
the development priorities to be pursued by these lOs. One can also 
observe that evaluations within IOs are likely to be as political an activity as 
they are within state institutions by virtue of the fact that all IOs are created 
by states to pursue shared interests and objectives. 


In the 21st century, programme evaluation has continued to evolve, and 
its qualitative value in development and policy practice has become 
augmented. Gelmon, Foucek and Waterbury (2005, p. 2) assert that 
programme evaluation has evolved and typically seeks to address the 
following questions vis-a-vis development policies, programmes and 
projects: 


e Is a development intervention (policy, programme or project) achieving 
its goals and objectives? Why or why not? 

e Is a development intervention achieving its intended impact vis-a-vis 
the intended beneficiaries? What is the nature of this impact (intended 
or unintended)? 

e Has the development intervention been implemented efficiently? 

e Is the development intervention responding to the development needs 
or interests of the intended beneficiaries? 


Thus, the qualitative approach of programme evaluation has enabled the 
practice to be able to explain the cause-effect relationship between 
development interventions and their intended beneficiaries. As mentioned 
in earlier sections, programme evaluation as a practice and discourse has 
been exported into Africa by IOs. An added dimension is that the standard 
development indicators measured by evaluations have also been shaped 
by these lOs rather than the African populations that are the targets 
of development interventions and who should therefore define what 
development means in their varied development contexts. The subsequent 
sub-themes provide an overview and a critique of the role IOs play in 
driving the evolution of programme evaluation practice globally and within 
Africa specifically. 


The impact of the Organization for Economic 
Co-operation and Development on programme 
evaluation practice 
In 1991, the OECD-DAC adopted five evaluation criteria, which became the 
standard objects of measurement for evaluations internationally. The original 


five OECD-DAC criteria are relevance, effectiveness, efficiency, impact and 
sustainability. In 2019, the OECD-DAC added one new criterion - coherence 
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(OECD 2019, p. 3). Collectively, the six OECD-DAC criteria are internationally 
recognised as key objects of measurement for evaluations. When conducting 
an evaluation, the re/evance criterion guides the evaluator to assess or 
examine the extent to which a development intervention (policy or 
programme) is responsive to intended beneficiaries’ needs. Coherence 
examines the alignment compatibility of the development intervention with 
other interventions in the same context (country, sector or institution) and 
compatibility with regional and global development agendas. For its part, 
effectiveness as a unit of measure in an evaluation seeks to determine the 
extent to which a policy, programme or project has achieved its intended 
objectives and medium-term changes vis-a-vis a target beneficiary. Efficiency 
focuses on whether resources are used in an economical manner to 
implement intervention activities and deliver the intended results (outputs 
and outcomes). When an evaluation measures impact, it is essentially 
assessing the extent to which a development intervention has contributed to 
intended and unintended long-term changes vis-a-vis targeted beneficiaries 
and their broader community or environment. Lastly, an evaluation that 
measures an intervention’s sustainability essentially examines the extent to 
which the benefits of an intervention have continued or are likely to continue 
after the intervention has stopped (OECD 2019, pp. 7-12). 


While the OECD-DAC criteria are important in aiding an evaluation’s 
judgement of the merit or worth of an intervention, this criterion does not 
adequately and explicitly emphasise equity. As mentioned in the conceptual 
framework, equity is important in the Global South, where colonialism 
structurally conditioned the pace of development across the two 
geopolitical regions (Global North and Global South). Because of the 
dominance of neoliberal conceptions of development globally, an evaluator 
and an evaluation commissioner who have no appreciation of equity are 
likely to ignore a development intervention’s contribution to equity issues 
(i.e. the extent to which an intervention improves the socio-economic 
opportunities of vulnerable or marginalised groups [i.e. females, youth and 
people living with disabilities], particularly their ability to earn living wages 
and have access to the means of wealth creation such as access to finance, 
land and social capital). 


In the African continent, the pressing development challenges are: 
poverty; inequality; political instability and insecurity (including civil wars); 
lack of access to quality education, health care and technology; energy 
insecurity; infrastructure challenges; food insecurity; and youth deprivation 
and discontent. A key question to ask is: has or is programme evaluation 
in Africa helping to address these aforementioned pressing development 
challenges through relevant inputs in public policy-making, resource 
allocation and government decision-making? For African evaluation 
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practice to be relevant and provide key advice to public policy-makers, 
evaluators in Africa need to have an adequate understanding of what 
constitutes development in different African nations and communities, 
particularly what constitutes equitable development. African evaluators 
who lack such an understanding end up measuring the wrong indicators 
and writing evaluation reports that do not proffer recommendations that 
can enable policy-makers to adopt policy actions that can genuinely 
eradicate poverty, inequality, youth unemployment and deprivation, 
terrorism and other development challenges on the continent. Thus, 
relevant evaluation practice requires Afrocentric and equity-based 
conceptualisations of development. Evaluation in Africa should therefore 
strive to provide African solutions to African developmental challenges. 


The African Evaluation Association (AfrEA) has attempted to infuse 
African dimensions to evaluation discourse and practice in Africa. The 
AfrEA (2021, p. 5) promotes five evaluation principles to be internalised by 
evaluators in Africa: 


e Evaluation that is inclusive, participatory and that empowers Africans. 

e Evaluation that is technically robust and culturally responsive. 

e Evaluation practice that affirms and promotes human rights and 
addresses power imbalances between evaluation participants, evaluation 
commissioners and the evaluation team. 

e Evaluation that draws from African indigenous knowledge as well as 
knowledge and evaluation practices from across the world. 


The AfrEA and other African evaluation stakeholders (evaluation 
associations, evaluators and scholars) have attempted to develop and 
promote the Made in Africa Evaluation (MAE) approach. The MAE approach 
is meant to be a coherent body of African evaluation approaches and 
principles that draws from African indigenous knowledge systems (IKSs), 
African development indicators and African development values (Dlakavu, 
Mathebula & Mkhize 2022, p. 5). Even though MAE has captured the 
imagination and attention of the African evaluation landscape, this 
emerging approach has not translated into African evaluation practice. 
In order for MAE to be adopted as a complementary or alternative 
evaluation approach and for its methodologies to be adopted in evaluations, 
there needs to be a popularisation of decolonisation ideology and for 
African continental and sub-regional institutions and governments to also 
champion this Africa-centric evaluation paradigm and its proposed 
methodologies. Currently, programme evaluation in Africa continues to be 
another discipline and practice that mirrors Global North monopolisation 
of knowledge in the global political economy. Global North monopoly over 
evaluation discourse and practice is sustained by ECD thinking and 
practices, which is discussed next. 
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Evaluation capacity development in Africa: 
Perpetuation of Western evaluative thinking 
or a move towards transformation and equity 
in evaluation? 


Table 3.1 outlines select formal evaluation course offerings at African 
higher education and training institutions. 


Table 3.1 shows only two explicit evaluation courses by the University of 
Ghana and the University of Cape Town, which explicitly focus approaches 
to evaluation that hold promise for placing a spotlight on matters of 
development equity and socio-economic transformation for marginalised 
social groupings in society. More African universities need to follow 
these examples from the aforementioned universities. For equity to be 
mainstreamed, it needs to be reflected in courses and not be ‘fitted’ in 
uncodified curricula or allocated hours in courses that last a year or two. 
This is how decolonisation has been silently removed or overshadowed 
because it has been an ‘implied’ part of social science, not an explicit one. 


A further notable issue that is raised in certain evaluation forums is the 
fact that evaluation appears a standalone course and divorced from 
associated courses such as public policy studies, public administration and 
development studies. This deprives evaluation students the opportunity to 
understand the utility of evaluation within development and public policy. 
Development and public policy are essentially the units of measurement 
for evaluations, and it is therefore interesting that over the years, evaluation 
studies have been divorced from development studies, public policy studies 
and public administration. Importantly, these other disciplines often have 
an explicit focus on equity; for instance, develooment studies’ focus on 
gender and development, children, youth and development, and grassroots 
development. Similarly, public administration and public policy studies 
tend to emphasise the importance of affirmative action legislation or pro- 
poor policies and social security, especially for previously marginalised 
individuals. If evaluation was still linked to these equity-focused disciplines, 
then equity would likely be a central focus. As things stand, evaluation 
studies appear to be focused on technical evaluation competencies such 
as research and evaluation methodologies, evaluation frameworks, 
principles and approaches. This focus is at the expense of being conscious 
of pressing development issues in Africa, such as equity issues and other 
development issues which evaluation practice should also seek to eradicate, 
such as youth discontent, political instability, the marginalised population 
groups, energy insecurity, food insecurity and the like. Evaluation studies in 
Africa therefore continue to borrow ‘blindly’ from Western programme 
evaluation studies, which renders African evaluation studies as yet another 
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TABLE 3.1: Monitoring and Evaluation qualifications in selected African higher education and training 


institutions. 


Higher education 
institution 


Formal evaluation 
qualification offering 


Standard modules 


Focus on equity or 
African evaluation 
approaches 


Africa Training Institute,’ 
online 


The Open University of 
Tanzania,* Tanzania 


Makerere University," 
Uganda 


University of Ghana,® 
Ghana 


Diploma in Monitoring 
and Evaluation 


Master of Arts in 
Monitoring and 
Evaluation 


Master of Public 
Health Monitoring and 
Evaluation (MPHME) 


Master of Science in 
Public Health Monitoring 
and Evaluation 


Introduction to M&E 


Project Planning and 
Implementation 


Evaluation Types and 
Modules 


Community-based 


Participatory Research 


Standard M&E methods, 
techniques and practices 
adopted by international 
organisations 


M&E concepts and 
theories 


Programme Design and 
management 


Monitoring principles, 
methods and 
approaches 


Statistical methods 
Programme evaluations 
M&E systems 


Research methods 
Biostatistics 


Research methods 
Fundamentals of M&E 


M&E of Gender 
Integrated Programs 


M&E of Public Health 
Nutrition 


M&E of Social Programs 


The community-based 
participatory research 
focus in one of the 
modules is an example 
of how to advocate for 
community-based M&E 
that is equitable and 
captures people’s voices 
in development and 
evaluation 


No equity/community/ 
grassroots M&E 
methods listed 


No module on equity 
in M&E 


University of Ghana’s 
M&E qualification is 
the most explicit of the 
sampled Universities 
here when it comes 

to codifying equity- 
focused M&E modules 
such as: 


1. M&E of Gender 
Integrated Programs 


2. M&E of Public Health 
Nutrition 


3. M&E of Social 
Programs 


Table 3.1 continues on the next page> 


3. https://www.africatraininginstitute.org/diploma/diploma-in-monitoring-evaluation/. 


4. https://www.out.ac.tz/ma-me/. 


5. https://sph.mak.ac.ug/academics/master-public-health-monitoring-and-evaluation-mphme. 


6. https://www.ug.edu.gh/sites/default/files/documents/SPH%20Msc%20M%26E%20Brochure%2010-05-2018% 


5B218%5D.pdf. 
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TABLE 3.1 (cont.): Monitoring and Evaluation qualifications in selected African higher education and 


training institutions. 


Higher education 
institution 


Formal evaluation 
qualification offering 


Standard modules 


Focus on equity or 
African evaluation 
approaches 


University of the 
Witwatersrand (South 
Africa) 


Stellenbosch University 
(South Africa) 


University of Cape Town 


Postgraduate Diploma in 
Public and Development 
Sector Monitoring 

and Evaluation’ and 
Master’s Degree in 
Management of Public 
and Development 
Sector Monitoring and 
Evaluation 


Postgraduate Diploma 

in Monitoring and 
Evaluation Methods® and 
Master of Philosophy 

in Monitoring and 
Evaluation 


Master of Philosophy in 
Programme Evaluation? 


Evaluation 


Managing the M&E 
practice 


Monitoring systems 


Public finance and 
performance budgeting 


Principles and 
paradigms of evaluation 
studies 


Clarificatory evaluation 


Process evaluation and 
programme monitoring 


Statistical and 
qualitative analysis 
methods for evaluation 
studies 


Evaluation report 


Principles in programme 
evaluation 


Statistics for evaluation 


Advanced quantitative 
evaluation design and 
analysis 


Alternate approaches 
for complex evaluation 


Qualitative methods in 
evaluation 


Alternate approaches 
for complex evaluation 
Ci.e. Africa-centric 
evaluation) 


Monitoring using 
programme theory 


Research design for 
impact evaluation 


Source: See footnotes for each university’s qualification offering. 
Key: M&E, monitoring and evaluation. 


No explicit mention of 
equity in evaluation 
nor a module on 
African approaches to 
evaluation 


No explicit mention of 
equity in evaluation 
nor express module on 
African approaches to 
evaluation 


There is a module on 
alternate approaches 
for complex evaluations, 
with a focus on Africa- 
centric and participatory 
evaluation that is 

crucial for ensuring that 
issues of development 
equity are tackled by 
evaluations 


7. https://www.wits.ac.za/course-finder/postgraduate/clm/pdm-monitoring-and-evaluation/. 


8. chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://www.sun.ac.za/english/pgstudies/ 
Documents/ARTS%20Faculty/Programmes/Postgraduate%20Diplomas/Postgraduate%20Diploma%20 
in%20Monitoring%20and%20E valuation.pdf. 


9. http://www.organisationalpsychology.uct.ac.za/orgpsy/Masters-in-Programme-Evaluation. 
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arena of Western knowledge reproduction and consumption rather than 
being an emancipatory discipline that is grounded on African development 
history, contemporary development experiences and indigenous African 
knowledge systems that should define development and how it should be 
measured. 


In addition to higher education institutions (universities), there are 
institutions that provide practice-based ECD services that are targeted 
at governments and non-governmental evaluation and development 
stakeholders. Among these ECD institutions are the Centre for Learning 
on Evaluation and Results, Anglophone Africa (CLEAR-AA), CLEAR- 
Francophone Africa (CLEAR-FA), Twende Mbele, African Parliamentarians’ 
Network on Development Evaluation (APNODE) and the West Africa 
Capacity-building and Impact Evaluation (WACIE) programme. For the 
purposes of this chapter, the focus is on whether these ECD stakeholders 
prioritise developmental issues of equity and transformation. International 
organisations with developmental mandates have shaped the ECD 
agenda, including the thinking and discourse. For instance, the OECD- 
DAC, which consists of 31 countries and multilateral develooment agencies 
that are the largest distributors of development aid, adopted an ECD 
strategy in 2010. The OECD-DAC (2010, p. 2) resolved to provide resources 
for evaluation training and conferences and develop evaluation frameworks 
with the aid of recipient countries (governments). One commendable 
aspect of the OECD-DAC’s ECD strategy of 2010 is that it recognised that 
ECD interventions should be targeted at building an enabling environment 
(e.g. an accountability and learning culture among political actors) and 
strengthening institutions implementing development interventions 
(OECD-DAC 2010, p. 3). Moreover, the OECD-DAC strategy emphasises 
the importance of country-driven ECD with support from top management 
in respective governments. 


One missed opportunity in the ECD strategy of the OECD-DAC is a 
mentioning of the importance of investing in evaluation frameworks 
centred on the principles of equity, diversity and inclusion. Such equity- 
based and Transformational Evaluation frameworks (i.e. evaluation terms 
of reference, evaluation plans, evaluation guidelines and policies) could 
drive evaluation stakeholders to measure the effectiveness of development 
interventions in improving the material conditions (living conditions) and 
economic opportunities of the poor, females, the youth, rural populations 
and people living with disabilities. Such equity-centred evaluation 
practice, especially by donors, would unearth key development lessons 
for future development aid that is likely to be genuinely emancipatory 
and drive sustainable socio-economic change in Africa and elsewhere in 
the Global South. Development aid in the 21st century cannot be narrowly 
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focused on poverty and disaster relief only, but should also seek to 
eliminate conditions that lead to poverty, such as: 


e lack of access to productive finance 

e lack of access to quality education (early childhood development, basic 
and higher education) that can lead to economic empowerment and 
prosperity 

e lack of access to quality health care from birth to old age 

e lack of access to quality basic services and infrastructure (electricity, 
water and sanitation, clean environment, roads) that attracts the 
concentration of economic activity in rural areas, towns and cities. 


Evaluation capacity development institutes and donor agencies should 
collectively promote evaluation discourse and practice that is rooted in 
measuring the development realities of the Global South by adopting an 
analytical framework that is grounded on equity and the lived realities of 
Africans. 


The World Bank is another key multilateral development and evaluation 
stakeholder that has contributed to evaluation discourse and practice 
globally. In its conceptualisation of ECD, the World Bank recognises the 
importance of using evaluation findings to improve the effectiveness of 
development outcomes and humanitarian relief. In essence, the World Bank 
views evaluation as an instrument of change, provided that the practice is 
undertaken by competent evaluators who are independent and use 
appropriate research methods in the evaluation process (Heider 2009, 
p. 2). While this principled posture on the value of evaluation is commendable, 
evaluation stakeholders in Africa need to contextualise evaluative thinking 
and approaches to the nuanced developmental history, dynamics and 
trajectory of this post-colonial continent. 


The African continent requires evaluation practitioners who understand 
that the design of evaluations in Africa needs to include culturally-responsive 
research methods and research questions that engage with the racial, 
gendered and spatial nature of development across Africa. For instance, 
if an evaluator is evaluating the effectiveness or impact of an economic 
development policy or programme in any African country, such an evaluation 
ought to ask the following questions: (1) how has the intervention improved 
the livelihoods of targeted beneficiaries? And (2) how has this intervention 
contributed to the social mobility of the beneficiaries and their dependents 
or families? This is because economic development is not only about creating 
jobs for targeted beneficiaries but also about improving people’s access to 
life-changing assets (land, property and finance) that can propel an individual 
and their families towards better living standards and a good life that is 
sustainable. Such a substantive understanding of what constitutes genuine 
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economic development should then translate into the development of 
substantive development indicators, evaluation criteria and evaluation 
questions that should result in valuable, credible and relevant evaluation 
reports that can be used as a basis for developing relevant economic 
development interventions that produce equitable development in Africa. 


Evaluation capacity development institutions ought to develop individual 
evaluation training programmes and institutional evaluation frameworks 
that are grounded on this central principle of equity. Without a genuine 
adoption of equity in ECD and evaluation practice in Africa, ECD and 
evaluation perpetuate the marginalisation of the poor, the youth, rural 
populations and females in Africa. Promoting equity in evaluation and ECD 
is also relatively easier to promote among political decision-makers and 
senior government officials because such a practice is aligned with the 
developmental mandates of governments. Current evaluation and ECD 
discourse and practice disproportionately emphasises technical aspects of 
this developmental practice while overlooking the key selling points: its 
value in the public policy cycle, development planning and budgeting. 
Moreover, a more holistic evaluation and ECD discourse and practice needs 
to emerge in Africa if programme evaluation is to assist policy-makers and 
development stakeholders in alleviating development inequality locally, 
nationally and regionally. 


Western universities have been at the forefront of updating ECD 
approaches and guidelines in the 21st century. Fitzpatrick, Sanders and 
Worthen (2011, p. 252) assert that evaluators should demonstrate cultural 
competence, meaning that their evaluation activities and findings should 
be based ona good understanding of the overarching context within which 
the evaluand (intervention being evaluated) and the evaluation is taking 
place. Such culturally-responsive evaluations are assumed to lead to 
appropriate and justified evaluation findings that are useful to the 
programme or policy decision-makers and other stakeholders. The author 
agrees with Fitzpatrick et al. because culturally-responsive evaluation 
reports may be of intrinsic value to policy or programme decision-makers 
because such an evaluation report would be informed by a sufficient 
understanding of the intervention context. Therefore, culturally-responsive 
evaluation practice prospectively offers relevant programme or policy 
recommendations that can lead to improved policy (programme) design, 
implementation and results. 


It is also the considered conclusion of the author that the development 
of context-responsive evaluation practice can be one of the drivers of 
equity-based evaluation practice and policy-making. Moreover, context- 
responsive evaluation practice can be a catalyst for the building of African 
evaluation practice that accurately captures the development experiences 
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of targeted African beneficiaries, thereby incorporating their indigenous 
knowledge of their development experiences and aspirations into the 
public policy and development planning and programming processes. This 
would help refine development programmes, plans and policies into change 
agents that are informed by diverse African voices and experiences of 
development, thereby offering plausible pathways on how beneficiaries’ 
lives can be improved through these development interventions. 


E Recommendations for transforming 
programme evaluation discourse and 
practice in Africa 


To ensure that African evaluation discourse and practice transforms into an 
equity-focused development instrument, the chapter proffers the following 
recommendations: 


e African universities as trainers of evaluation professionals, African 
universities should (re)integrate evaluation studies into development 
studies and public policy studies. Development studies offers extensive 
knowledge about what development entails theoretically, while public 
policy studies focuses on empirical solutions to development issues 
facing Africa and the world. Public policy and development studies 
also provide a historical and contemporary context for international 
and African development. From these two disciplines, evaluators in 
Africa can get to understand the historical and structural challenges to 
development, which helps with understanding how to measure 
development. The importance of equity in evaluation can be adequately 
emphasised if this discipline gains a philosophical, theoretical and 
empirical foundation from development studies and public policy 
studies. 

e Evaluation capacity development institutions, governments and 
donors should champion programme evaluation frameworks (policies, 
guidelines, indicators and evaluation criteria) that emphasise the 
importance of measuring the extent to which development programmes 
and policies promote equity for the marginalised social groups in society. 
For example, evaluation guidelines and criteria should focus on the 
extent to which a development intervention has created opportunities 
for livelihoods, access to finance and wealth creation for females, people 
of colour, the youth and people living with disabilities. Such a focus on 
equity criteria would provide substantive insights into the genuine 
effectiveness and impact of development interventions in Africa and 
elsewhere in the Global South. A further recommendation for ECD 
institutions and donors would be for these stakeholders to grapple with 
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the question of how to work with African governments and civil society 
to build equity-focused evaluation systems across development sectors 
of African nations. That is, how can we build equity-centred evaluation 
systems in social development sectors (i.e. health, education and social 
security sectors)? How can we build energy sector evaluation systems 
that can contribute to energy security for all people and groups in 
society? How can we build agriculture sector evaluation systems that 
can contribute to food security in a country? How can we build financial 
sector evaluation systems that improve access to finance for 
entrepreneurial, business and education-seeking females, youth, people 
of colour and people living with disabilities? A further question for 
African governments is: how can governments adopt evaluation as a 
permanent tool to measure the effectiveness of macroeconomic policies 
(i.e. public expenditure, taxation and monetary policies) in improving 
development outcomes? 

e Evaluators and evaluation scholars in Africa must champion the 
aforementioned principles of equity in evaluation and ECD discourse 
and practice. In their evaluation communities of practice and associations, 
African evaluation practitioners and scholars should continue advocating 
for decolonised evaluation discourse and practice that can highlight 
Africa’s continuing development challenges and the global political 
economy that continues to reproduce or maintain the dispossession of 
Africa developmentally. African evaluators, or evaluators in Africa, need 
to persist with the quest to mainstream decolonisation of development 
and evaluation discourse and practices, as that is a historic mission that 
cannot be outsourced to evaluators and institutions beyond Africa. As a 
result, the chapter argues that the AfrEA and other national evaluation 
associations in Africa, as well as evaluators in Africa are the would-be 
advocates of Africa-centred evaluation methodologies that will highlight 
and recommend remedial actions for addressing key socio-economic, 
political and global challenges that limit African development potential. 


E Conclusion 


Evaluation in the Global South must strive to be an emancipatory discipline 
and practice and not reproduce inequality and Global North hegemony 
through unconscious evaluation discourse and practice. This chapter has 
reflected on the current nature of evaluation discourse and practice in 
Africa, arguing that there is scope for the advancement of a more equitable 
and context-specific evaluation discourse and practice on the African 
continent and elsewhere in the Global South. Such an equity-focused 
evaluation discourse and practice is to be informed by African and Global 
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South development thinking, experiences and aspirations (i.e. indigenous 
conceptualisations of what constitutes development and the indicators of 
development from African peoples’ perspectives). Through such African 
insights, equitable evaluation discourse and practice can be shaped by 
various evaluation stakeholders in Africa: universities, governments, donors, 
ECD institutions, evaluation scholars and practitioners, and evaluation 
communities of practice at continental and national levels. The chapter has 
outlined recommendations on how each of these evaluation stakeholders 
can help drive an equity-focused evaluation discourse, agenda and practice 
in Africa that can be emulated elsewhere in the Global South. The author 
argues that an equity-focused evaluation posture, discourse and practice 
can contribute to the alleviation of inequality, poverty and marginalisation 
in the African political economy and the global political economy at large. 
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E introduction 


There is no doubt that the world is characterised by inequality and 
inequities. Inequality continues to manifest itself at political, business, 
international organisation, educational and country levels. A Marxist 
analysis of class offers a better understanding of how inequality emerged 
and has sustained itself even during contemporary times. The social 
relations of production contributed to the emergence and re-emergence of 
inequality (Onwubiko & Okooroafor 2016). This is epitomised by the 
relationship between the bourgeoisie and the proletariat. The unbalanced 
relationship where the bourgeoisie owns the means of production and the 


How to cite: Chirau, TJ & Saidi, U 2023, “Quis custodiet ipsos custodes?” Un(masking) inequality by 
international institutions’, in S Masvaure, TJ Chirau, T Fish & C Morkel (eds.), Equitable Evaluation: Voices 
from the Global South, Evaluation: African Perspectives, vol. 1, AOSIS Books, Cape Town, pp. 69-88. 
https://doi.org/10.4102/aosis.2023.BK459.04 
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proletariat offers labour to receive a salary hardly keeps the proletariat out 
of poverty. 


It is by understanding this asymmetrical relationship that inequality 
stubbornly manifests itself in every soace where human agents interactively 
connect. The class struggle is replicated by the international institutions, 
albeit in new forms. Institutions such as the United Nations (UN) and its 
agents, the World Bank and International Monetary Fund (IMF), are no 
exception in perpetuating inequalities, particularly in regard to the Global 
South. This is evident in how the Global North relates to the Global South 
in multilateral organisations. In that light, Onuoha (2008) argues that: 

The co-operation of various states that constitute the international system is 

important for the survival of the system. Unfortunately, some states are more 

powerful than others and these powerful states now dominate and determine the 
operation of these organizations. In most cases these international organizations 


openly function to protect the interest of the rich countries which wield more 
economic power. (p. 38) 


In light of this, international organisations - most of which function as 
funders and promoters of policies on development and are openly biased 
- it is argued that inequity and inequalities are initially traced to the work 
of such organisations. The aim of this chapter, therefore, is to expose the 
hidden sources of inequality manifesting in the evaluation practice and 
space. This chapter hypothesises that the main nucleus of inequity and 
inequalities existing in monitoring and evaluation (M&E) practices are 
traced to the operations and attitudes of international institutions towards 
the Global South because these provide advice and recommend 
interventions based on knowledge gained from M&E practices aimed at 
addressing poverty (and other challenges) facing Global South countries. 


As the African region pushes and seeks to develop the Made in Africa 
Evaluation (MAE) agenda, it must reflect not only on how evaluation has 
been under siege in Africa but also that institutions and organisations 
have been using systems and frameworks that force them to produce 
positive outcomes within contexts that do not support imported epistemes. 
Of concern in this chapter, however, is suggesting ways of establishing the 
best ways of working with Global North-induced M&E frameworks. It is 
hoped that this will minimise the promotion of institutional inequalities, 
which often implicate institutions and organisations in the process. 


It is crucial to study and map variabilities in institutional or organisational 
(those whose business is M&E) inequalities. This will not only allow 
refocusing but also an appreciation of how macro- and micro-level 
‘political, economic, and social conditions shape the distribution of more 
proximate determinants’ (Bakhtiari, Olafsdottir & Beckfield 2018, pp. 1-2) 
of inequities and inequalities. There exist broad institutional arrangements 
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prone to inequities and inequalities as institutions provide an interface 
and create relations between groups and professional agents. It is against 
this view that the subscription to the idea that inequality stubbornly 
manifests itself in every soace where human agents interactively connect 
to solve challenges that face humanity. If, in political, business or 
educational spaces, to name a few, inequalities exist and continue to 
baffle societies, it also means evaluation spaces are not unique. The 
conditions there are worse than those institutions operating in Africa, 
wherein global evaluation matrixes come into play against the prevailing 
environmental and contextual factors. 


When ‘evaluation’ is taken as an institution, apart from it being a practice, 
it means inequalities further manifest ina variety of ways in the administrative 
institution and as a concept or discipline. Thus, this chapter discusses and 
provides illustrative instances of institutional ‘evaluation inequalities’ in 
southern Africa, hoping to enhance MAE by minimising hurdles along the 
way. This chapter further reveals that many inequalities exist because of 
the superimposition of ontological, axiological and epistemological aspects 
defining the operationalisation of evaluation in Africa. It is argued that 
locating and placing Indigenous Knowledge Systems (IKSs) at the centre 
may help close the inequality gaps observable in evaluation spaces and in 
the discipline itself. 


Significance and research questions 


Research focusing on inequality is quite substantial globally, relating also 
to inequalities in Africa. Several studies demonstrate, with the greatest 
magnitude, the degree of inequality and inequity in most societies in the 
world (Ahmad, Haque & Islam 2022; Costa 2010; Dutt & Mukhopadhyay 
2009; Hayes, Introna & Kelly 2018; Hujo 2021; Levy et al. 2020; Silva Filho et 
al. 2023). Much of this research links inequalities to the attitude and 
relations of the Global North and Global South. It is reflected that inequality 
has a very long history of being propelled by institutions undertaking 
developmental practices and processes in Africa (Hayes et al. 2018; Hujo 
2021; Levy et al. 2020; Silva Filho et al. 2023; Yekini & Yekini 2020). Gaps in 
research exist that fundamentally call for novel strategies to institute 
paradigm shifts in the nature, structure and practices of M&E in Africa. 


This chapter, therefore, fills in this important lacuna in the African M&E 
research by drawing and calling for the need to feed from novel African 
epistemologies to address inequalities resulting from evaluation practices, 
notably driven by international organisations operating in the Global South 
Cin this case, Africa). This chapter also makes an analytical contribution to 
M&E research by recognising and appreciating the significance of both 
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structure and agency in Global South countries, specifically with reference 
to MAE and IKSs. The Global South countries are not passive victims of the 
machinations of Global North evaluation practices but are active agents 
that aim to emancipate themselves from the chains of Global North 
evaluation imperialism and practice. 


The analogy expressed by Besley and Robinson (2010, p. 655) that ‘Quis 
Custodiet Ipsos Custodes’ regards the question of ‘who guards the guards’, 
and this is alluded to in this chapter to intimately demonstrate its 
connections to evaluation practices in Africa, where development 
institutions and their agencies identify themselves as ‘guarding’ the Global 
South against poverty, underdevelopment, pandemics, inequalities, health 
care and food security. This gives rise to the recurring questions about why 
the greater part of the Global South remains underdeveloped despite the 
billions of dollars in funding and the insurmountable presence of 
international institutions and their agencies working for years in the South 
to eradicate poverty, among other challenges. 


In that light, and fundamentally so, this study’s main objective is to 
unearth the sources of inequity and inequalities that manifest in the 
evaluation practices by a cohort of international institutions providing 
advice on interventions aimed at addressing poverty and other challenges 
facing humanity in the Global South. The preoccupation is with returning to 
the source in order to appreciate how inequity and inequality appear to be 
‘planned’, based on knowledge obtained from evaluation reports from 
which recommendations witness mechanisms that define how M&E is 
practised. Sustainable interventions can be suggested only with the full 
knowledge of the source of inequity and inequality. In other words, the 
perpetuation of ‘globalised’ M&E practices divorced from considerations of 
indigenous epistemes is perpetuating underdevelopment (Rodney 1972). 
To achieve this, therefore, the following research questions guided this 
chapter: Why are evaluation practices Global North-based? To what extent 
have evaluation practices by international organisations helped African 
countries manage and develop their own evaluation systems? Who makes 
decisions when it comes to the management and commissioning of 
evaluations? What are the available options to bring about equality 
epistemologically? 


E Methods 


The authors used systematic search methods to collect relevant literature. 
This chapter relied mainly on secondary data found in peer-reviewed 
journal articles and reports accessed from various online repositories. The 
chapter also benefited immensely from the authors’ experience in working 
with indigenous epistemes, M&E space, strengthening M&E systems of 
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English-speaking countries and with central government institutions that 
provide oversight and coordination of public sector M&E systems. This 
chapter further benefits from the authors’ experience in working with 
multilateral organisations supporting African countries in the development 
sector. A variety of search items were used, including inequality in 
multilateral organisations’, inequities, MAE and IKS. 


Theoretical foundation 


Research shows how inequalities are linked to developmental disparities 
caused by some questionable relations between the Global North and 
Global South (Hancorck 1989; Hujo 2021; Mugge et al. 2018). Most agencies 
or institutions headquartered in the Global North fund developmental 
programmes designed to eradicate poverty in the Global South. However, 
by default, design or otherwise, developmental interventions have resulted 
in inequity and inequalities. For instance, in the 1980s and 1990s, Global 
South countries (particularly those in Africa) that implemented the IMF- 
and World Bank-prescribed structural adjustment programmes (SAPs) 
(Hancorck 1989) suffered serious economic recessions. Experiences of the 
implementation of SAP (locally known as Economic Structural Adjustment 
Programme - ESAP) are still alive in the memories of most southern African 
citizens whose countries adopted the adjustment programmes. The ESAP- 
linked policies resulted in the cutting out of state subsidies to low-income 
groups, causing widespread retrenchments, thereby escalating the cost of 
living and lowering real wages (Dube 2010). Tamukamoyo (2009, p. 97) 
argued that the ‘informal economy came into its own’ in the 1990s because 
of the negative impacts of ESAP, with backyard industries and petty trading 
sprouting everywhere as workers and families adopted a range of livelihood 
strategies. Close to 28 countries that implemented SAPs had their 
economies shrink (Onwubiko & Okoroafor 2016). As a result, international 
organisations intervened, mostly through programmes and projects 
(hereafter referred to as interventions) cutting across all sectors of the 
economies, notably, education, agriculture, politics and health. 


As a condition to continue receiving support from UN agencies, the IMF 
and the World Bank, target countries are supposed to receive, monitor and 
evaluate the programmes. In analysing problematic issues around aid 
management in Africa, Hancorck (1989) noted the ‘claim’ that developmental 
institutions make regarding their main role. This was indicated as providing 
and supporting policies, attitudes, behaviours and engagements that must 
ultimately empower beneficiaries in the target country (Hancorck 1989). 
Sadly, this modus operandi has been subtly inducing inequality and inequity. 
There is an imposition of the systems of M&E on beneficiary countries, 
for instance. 
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Theoretically, therefore, the foregrounded is pinned on the Dependency 
Theory (DT). The DT gained traction in the 1960s as a critique to 
modernisation and trickle-down economic theories. It was originally 
applied to appreciate inequalities existing between the developed (Global 
North) (the core) and developing countries (Global South) (periphery) 
with regards to underdevelopment and industrialisation (Rodney 1972). Its 
application was in Latin America, with scholars such as Paul Prebisch 
leading the conversation. 


To explain underdevelopment in other parts of the world, such as Africa, 
the dependency hypothesis has been employed in several different ways 
(Rodney 1972). The DT is analytically useful to this study because of its 
sensitivity to power relations between the Global North and Global South. 
To this end, it is therefore a useful tool to pinpoint and understand how 
development agencies perpetuate inequality despite other 21st-century 
challenges. This is despite the constraints of the DT and its problem of 
generalisation and ethnocentrisms (Farny 2016). 


Dos Santos (1970, p. 231) defines dependency as ‘a situation in which 
the economy of certain countries is conditioned by the development and 
expansion of another economy to which the former is subjected’. The DT 
separates the world into core and periphery economies. Dependency 
theorists, on the other hand, analyse ‘systems’, with a focus on how 
the global capitalist system contributes to the underdevelopment of the 
periphery (Kay 2011; Sekhari 2009). Tausch (2010) argue that: 

[...] poverty and ‘backwardness’ in the ‘periphery’ and semi-periphery are 

caused by the very peripheral or quasi peripheral position that these nations or 


regions always had in the international division of labor since the beginnings of 
the world system in 1492. (p. 468) 


The main pre-conceptions of the theory consist of (Namkoong 1999): 


1. A lack of development, as opposed to un-development, is the 
premeditated extraction of resources from the periphery nations in 
order to benefit the core nations. 

2. Periphery countries are poor because of being involuntarily incorporated 
into the international division of labour, where they serve as producers 
and suppliers of raw materials or reservoirs of cheap labour. 

3. Active co-operation between dominating powers and local elites who 
have similar interests maintains resource diversion. 


Furthermore, Tausch (2010) brings another perspective and summarises 
the key characteristics of the periphery as the main features of the periphery 
and semi-periphery countries: 


[...] a high penetration by foreign capital, a heavy c from the leading countries, 
the overall subordination of the productive capacities of the country towards the 
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interests of the evolving international division of labor, [and] the concentration 
of exports on a few commodities and recipients. (p. 469) 


The economic, political, military and ideological arenas are only a few 
where the unequal and exploitative interactions between the Global North 
and the Global South can be seen (Galtung 1971). It is on these bases that 
dependence in the ‘world system’ is used to examine, understand and 
explain global inequalities (Farny 2016). However, with the increased 
globalisation, it is contemporarily challenging to hold the notion of 
inequality based on neo-colonialism, integrated world systems and 
economic dependency (Randall 2004). The rise of the so-called Asian 
tigers is a contradiction to the notion of an integrated world system. 
However, the theory has its loopholes and is subject to criticism. For 
instance, the DT fails to take into consideration the progress of some 
traditional periphery countries and also implies that it was impossible to 
achieve development within the context of global capitalism (Amsden 
2003). Critics suggest that an integrated global economy could lead to 
economic improvement (Sanchez 2003). Kvangraven (2021) notes that the 
core tenet of the DT is not in conflict with the economic transition of the 
Asian tigers, rendering the theory weak in such cases. It is, therefore, 
essential to ask: ‘Does this kind dialectical relationship exist and manifest in 
the evaluation practices of institutions and or organisation?’ This chapter 
does not provide direct answers but stimulates debate while seeking a way 
forward to reduce the dependence of the Global South on the Global North. 


E Programme evaluation 


Programme evaluation is defined as ‘judging the worth or merit of 
something or the product of the process’ (Scriven 1991, p. 139). Stakeholders 
and policy- and decision-makers desire that programmes serve the 
intended goals. Monitoring and evaluation is a growing profession across 
Africa, and this is perhaps indicative of the acknowledgement of the value 
that the discipline and profession hold. In Africa, universities are playing a 
significant role in capacitating individuals desiring to pursue careers in 
M&E. In turn, individuals play an important role in government and non- 
governmental institutions working to track the progress - or the lack 
thereof - of different interventions, be it programmes, projects, policies or 
plans, among others. These universities in Africa have a strong reputation 
for offering M&E programmes pitched at different levels - which is the case 
for some, but not all. Countries such as Kenya, South Africa, Uganda and 
Zambia have standalone M&E offerings at certificate, diploma, postgraduate 
diploma, MA and PhD levels (CLEAR-AA 2019). Other training is offered by 
the Voluntary Organisations for Professional Evaluations (VOPEs) and 
government institutions, but they are not credit-bearing (CLEAR-AA 2019). 
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As such, critical questions are raised as to what influence the Global North 
has on African universities and their courses in M&E? Evaluation imperialism 
is the source of the current domineering of evaluation approaches, theories, 
frameworks and practices. The USA, Canada and Britain are usually cited as 
authors of evaluation imperialism (Mouton et al. 2014). Sadly, the African 
education systems responsible for embedding evaluation knowledge are 
still ‘captured’ by the Global North knowledge economy - that is, Knowledge 
dealing with M&E. If the evaluation educational systems remain as-is, 
influencing Africanised evaluations may remain unattained. Renowned 
global evaluation icon theories and approaches remain the staple for any 
African scholars or M&E trainers and students as many of these icons drive 
debates around topical issues. Anecdotal evidence from people across the 
continent, who have obtained different qualifications (certificates, diplomas 
and degrees of all levels) indicated that the influence of the Global North 
knowledge, including theories and case studies that are taught at 
universities, cannot be underestimated. 


There is little to no mention of the African methods taught in M&E 
classes, raising critical questions about whether African scholars, academics 
and evaluators are on the right trajectory towards Africanising the M&E 
courses in universities in Africa, for instance. As the tide of wanting rises to 
have evaluations, which are sensitive to African culture, norms, values and 
knowledge, there are bottlenecks acting as buffers. One of the bottlenecks 
relates to universities practising isomorphic mimicry, thereby perpetuating 
inequality in the process. What is also visible is inequality between African 
universities that are offering M&E. Aspiring scholars tend to favour certain 
universities compared to others based on the richness of the curriculum. 


The evaluation ‘elephant’ in the room 


There is a need to face and directly address ‘the elephant’ in the ‘small’ 
evaluation room, that is, MAE. Available research on MAE is still in its infancy 
with regards to addressing the ‘elephant’ in the room adequately and more 
directly and is still scraping unceasingly around the ‘room’. This implicit 
rather than explicit attitude stems, in part, from the epistemic positioning 
of evaluation, generally dominated by the Global North in terms of 
frameworks, systems and approaches. We claim, therefore, that there is a 
dearth of studies exposing or reporting on notions of inequality and equity 
within developmental agencies and institutions. Part of the reason could be 
the existence of what scholars have termed ‘epistemic inequality in a 
developmental perspective’ (Silvia et al. 2022), whichis a form of ‘interaction 
[...] aimed at producing a belief about some relevant aspects of the world, 
present or absent, past or future and at assessing its degree of epistemic 
trustworthiness’ (Silvia et al. 2022, p. 1). 
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In evaluation terms, institutions and development agencies assume the role 
of an agency, and their work, best described by Silvia et al. (2022), involves 
producing reports that are used to initiate a dialogue intended to influence 
the actions and behaviour of the objectified subjects. The relationship 
between evaluators and their ‘clients’ often overshadows reverse evaluation. 
This is premised on the understanding that (Silvia et al. 2022): 

Human societies progressed because of the ability to cooperate epistemically. 

Individuals share their beliefs, transmit reliable information, and contribute to 

the establishment of a collective knowledge network. This collective knowledge 


allows each individual to come into contact with parts and aspects of the world 
that would otherwise be inaccessible to them. (p. 1) 


It follows, therefore, that M&E hovers into the realm of collective knowledge 
where one expects some degree of equity on both sides whether one is 
evaluating or being evaluated; whether a programme is being evaluated or 
the programme outcomes are forcing reverse evaluation in an endeavour 
to create an equitable platform for all parties (recipients and implementors). 
It becomes a human activity, prioritising ‘epistemic’ cooperations, and such 
cooperations often face obstacles such as social and epistemic injustices 
which Silvia et al. (2022) term as: 

Breaks [/n] the cooperative chain [that] harms society and [eva/uators] in equal 

measure. Society is harmed because the social body ceases to have access to 


relevant information, while the [eva/uator] is harmed because they lose their 
legitimate right as a subject of knowledge. (p. 1) 


Evaluation processes, therefore, are assumed to involve a balancing act 
while providing recommendations designed to neutralise epistemic 
injustices that, in the broadest sense, M&E works to stifle. It follows, 
therefore, that internal systems of institutions mandated to ‘neutralise 
epistemic injustices’ operate with internal equities and equalities. There are 
no better ways to explain this systematic creation of dependence by the 
Global North than the modus operandi shown in M&E today. In this chapter, 
reverse evaluation is understood as evaluation by independent parties or 
agents who often get evaluated in the process. It is an attempt to re-shift 
focus and place similar evaluation systems and frameworks and use them 
to identify forms of inequity and inequality within institutions deemed 
highly knowledgeable in aspects of underdevelopment, and not only to 
address the ‘credibility’ of the process and agent. 


The social inequalities characterising contemporary societies (ldemudia 
2017; Yekini & Yekini 2020) cause unequal access to social amenities or, in 
evaluation terms in this context, cause professionals to fail to fully undertake 
their professional evaluation duties and access the needed resources for 
their work. This, ultimately, regards ill-treatment of the professionals as 
agents of knowledge. It is generally taken that the forms of global economic 
structures are responsible for inequalities (Hancorck 1989; United Nations 
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Children’s Fund [UNICEF] 2013). As organisations and institutions interact, 
the reproduction of inequalities is uncovered in one way or another (Ahmad 
et al. 2018). 


There is no way that institutions of ‘democracy’ and ‘accountability’ 
will remain unaffected by the global economic inequalities within an 
environment where socio-political instability is threatened daily by 
policies, some of which are ‘imported’ or suggested by the Global North. 
Amis et al. (2018, p. 1131) observe that developmental organisations and 
institutions, as agents interacting with various groups, ‘are heavily 
implicated in the rising levels of global inequality’. While Amis et al. (2018) 
discuss the aspects of inequality and its detrimental effects in the various 
regions of the world at great length, the exposé provides enough 
justification for interventions of one form or another by agencies and 
non-governmental organisations (NGOs) hoping to level inequalities. 
However, internal operations and structural aspects are ignored, although 
they recognise that institutions and organisations are sites of creating 
and perpetuating inequality. 


Despite this, reference to the nature of inequalities is placed elsewhere, 
wherein it is indicated that the global prevalence of inequalities and their 
trends do not imply that ‘the international institutions are to blame for this 
increase in inequality’ (Dutt & Mukhopadhyay 2009, p. 324). Despite the 
‘epistemic silence’ demonstrated by the deafness on reference to internal 
structures and bases of operation by the international institutions in this 
case, one reads an implicit hint in institutional structures that perpetuate 
inequity and inequality, especially in operationalising M&E. For instance, 
Dutt and Mukhopadhyay (2009, p. 328) indicate that the Bretton Woods 
Institutions (BWI)’s voting (deemed problematic) and the World Trade 
Organization’s (WTO) decision-making systems and manner of dispute 
resolutions ‘load the dice against poor countries in their interactions with 
rich countries’. Regarded as the ‘world government’, the UN is therefore 
under the spotlight where the current structure, for instance, of the Security 
Council, is such that five countries are ‘permanent’ members, notably 
China, France, Russia, the United Kingdom (UK) and the United States of 
America. The then president of Zimbabwe, Robert Mugabe, is on record for 
unceasingly calling for reform in the UN Security Council against the notion 
that (Campbell 2016): 

The UNSC is more involved in Africa than in any other region, and many Africans 


feel it is acutely unjust that none of the permanent members are from the 
continent. (n.p.) 


It is no accident that such a call is made around aspects of global politics 
from other international institutions such as the IMF, World Bank and 
other agencies that ultimately operate in accordance with the 
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recommendations of global political policies or calls. This is an instance of 
recreations of inequalities and inequity within institutions deserving 
serious debates on reverse evaluation, especially in matters that involve 
M&E in Africa. Global inequalities, therefore, have a direct influence on 
M&E at a micro level as the entire cycle of M&E ends up being aligned 
with the demands of funders who, in this case, are the development 
agencies controlling said institutions. 


The development space has, over the years, come to be defined by the 
dictates, influence, work and expansion of donor organisations. This 
explains the conclusion presented by Hayes et al. (2018, p. 1), that 
‘governments and other donor organizations increasingly require NGOs to 
account for their impact’. In this case, impact assessment (IA), otherwise 
known as knowledge management, is linked to evaluation processes. Given 
that this isan evaluation aspect, governments and other donor organisations 
would naturally desire NGOs to provide such interventions, placing NGOs 
on unequal footing with governments or the clients they serve, rendering 
the IA donor-centric NGOs exposed. It is noted that there is a top-down 
reporting structure by NGOs or other agencies as such reporting structure 
‘leads to disempowerment among NGO workers’ (Hayes et al. 2018, p. 2). 
For instance, the global village today witnesses multicultural and 
multilinguistic set-ups or contexts of operations by NGO workers. The use 
of English or some other ‘colonial’ language, for instance, excludes some 
employees in the process, thereby depriving important professionals of 
participation in and access to reports. 


The fact that NGO workers have been ‘mimicking’ Western knowledge, 
such as reports viewed as superior to non-Western knowledge-sharing 
through storytelling, means that local workers mimic Western forms of 
accountability. This is another instance of how inequality is realised. In fact, 
the process of mimicking Western forms of accountability while operating 
in non-Western contexts or settings is in itself evidence of the existence of 
inequity and inequality in these organisations, and research appears to 
avoid confronting this. Consequently, resources, time and effort are spent 
in ‘professionalising’ and meeting ‘donor’ or funder requirements ‘leaving 
them with less time to work with beneficiaries’ (Hayes et al. 2018, p. 3). 


Most international institutions in the development space or sector 
preoccupy themselves with developing interventions aimed at eradicating 
poverty in the world. Hujo (2021) noted that, for the past fifteen years, this 
preoccupation has been a characteristic of most institutions, but without 
the desire to address inequality. The 2030 Agenda for Sustainable 
Development of 2015 reconfigured the past developmental trends. This is 
evidence of continued declarations of undertaking the same old visions 
without solutions in sight. 
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Africa stands as the most vulnerable region and is presented as such in 
most of the evaluation reports. International institutions do not project 
themselves as being among the agents recreating inequalities. Studies 
show that international organisations merely ‘recommend’, in most cases, 
to governments and policy-makers the knowledge and policy directions 
required to eradicate poverty and inequalities, as these are believed to be 
responsible for underdevelopment. Failure to adopt the ‘recommendations’ 
or the failure of a programme is placed squarely on the heads of the 
governments, as it is understood in Eurocentric terms that African 
governments ‘dispose of less resources for equalising post-market 
distribution’ (Hujo 2021, p. 2). The World Bank ESAP adopted by most 
southern African countries in the 1990s is regarded in literature as having 
impoverished the populations of countries such as Ghana, Malawi, Zambia, 
Mozambique and Zimbabwe. 


What is missing in all this is the call for ‘reverse evaluation’, especially 
when one reads into Yekini and Yekini’s (2021) submissions that NGOs have 
been at the forefront of urging profit-making institutions to produce 
sustainable reports based on their operations. Because NGOs are mostly 
‘non-profit-making’, existing in as far as they provide their services to 
communities and individuals in an effort to address poverty eradication, 
homelessness and education for the less privileged, among others, this is 
thought not to injure the contextual environment. However, as NGOs 
operate in communities, the positive and negative environmental, cultural, 
economic and social activities need to be accounted for. 


Non-governmental organisations and other agencies often rally behind 
the ‘sustainability’ of programmes or interventions. Sustainability entails 
increased quality of life, wherein the poor can now afford basic socio- 
economic needs following amelioration of the consequences of negative 
environmental activities that usually arise from rendering the public good 
(Yekini & Yekini 2021). Indeed, it is not always the case that governments 
will efficiently function to helo communities and individuals living below 
the poverty line. Because inequalities exist in society, resulting in some 
groups failing to access socio-economic services to ease their situations, 
NGOs are vital in bringing interventions to address and promote inclusivity. 
The challenge, however, is the relationship or interface between NGOs and 
governments whose operations resultantly become ‘evaluative’ of the 
other, thereby exposing systems from both sides that leave the two 
institutions guilty of creating inequality and inequity. 


These discussions usually involve readily available evidence from the 
government perspective wherein NGOs produce evaluation reports 
pinpointing areas where the government may be found wanting and 
providing counsel on policy development. Sadly, the same cannot be said 
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of the NGOs, as what is awash are government rhetorical evaluative and 
political reactions that sometimes appear to take political-reactionary 
stances. In Zimbabwe, NGOs have been openly accused of harbouring a 
regime-change agenda by the Zimbabwe African National Union - Patriotic 
Front (ZANU-PF) government. The thinking is quite simple, in that NGOs, 
agents and donors are mostly funded by the Global North. The power 
dynamics that exist between the funder and the developmental institutions 
make it easy for one to assume that operations and programming will 
somehow be directed by the expectations of the funder. In this regard, 
programmes that insinuate or place beneficiaries to ‘question’ their own 
government, resist or demand ‘policy change’ are faced with the African 
governments responding harshly to regime change. Meanwhile, in 
Bangladesh, NGOs were accused of operating and promoting actions and 
behaviour which contradicted the teachings of Islam (Karim 2004). 


Ordinary people who have benefited from NGO programmes for a long 
time, for instance, may come to be baffled as to why their own government 
fails to undertake its duties, leaving the NGOs, for instance, to be what they 
‘know’ and rally against, even in terms of efficiency and the manner in 
which food may be distributed, health care initiatives implemented, 
employment (as locals are employed by the NGOs or hired in one way or 
the other and paid handsomely), education and other aspects. NGOs, 
agencies and other developmental institutions, hence, unintentionally 
promote inequity in society as those working directly with the NGOs 
benefit while others, perhaps in similar situations but located elsewhere 
geographically, may not receive the same. 


In some cases, a dependence syndrome is created, such as what has 
been seen in the Chivi communities of south-eastern Zimbabwe, an area 
characterised by prerenal droughts and floods (Madziyauswa 2018). In any 
case, the more the NGOs become operational and develop strong ties with 
local communities, especially rural communities, the more visible the 
message of continued failure by the government to provide social services 
to its people in general, let alone access the rural communities regarding 
developmental issues (only during election season does the government 
reach even the last cell of its political movement to canvas for votes). 


However, it does not follow that all is rosy when NGOs are involved or 
dominate a certain developmental space. In Bangladesh, Islam (2021) 
reported that in the case of the NGOs operating in that country for the 
provision of health care, while they were providing health care services 
according to their own policies, strategies and guidelines, the government 
institutions offered far better in service provision compared to some NGO- 
led institutions, save for ‘well-off families had good connections with NGO 
clinics, enabling them to obtain better health services than the poorer 
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families’ (Islam 2021, p. 1).However, generally, most NGOs had better 
services premised on better funding being availed. 


E Guarding the guards of development 


Applied and basic epistemology studies, mainly those that aim at 
modelling M&E, have encouraged the adoption and use of IKSs in all 
practices designed for human development in Africa. The strides that 
MAE has taken are beginning to be felt in a number of areas, mainly in the 
operations of organisations and in concerning themselves with research 
and development (Chilisa 2015; Chilisa & Mertens 2021; Mbava & Chapman 
2020). Carman (2011), Chamaki, Jenkins and Hashemi (2019), Chilisa 
(2015), Chilisa and Mertens (2021), Curry (2019) House (2015), Kabonga 
(2019), Mbava and Chapman (2020), Omosa et al. (2019), White (2009) 
and Winthereik and Jensen (2017) all agree that scholarship has called for 
IKS and development discourses to be taken back to the grassroots as 
well as foundational aspects of research wherein funders are now making 
strides to fund IKS-based research. Research on the African cosmos is 
now being called upon to be IKS-based, wherein IKS practitioners, 
knowledge holders and indigenous communities get to exercise equal 
control over the research conducted or in cases where members of such 
communities partake in the study. 


In South Africa, the IKS Focus Area Programme (FAP) is a notable 
development. When it was established in 2001, it promoted the ringfencing 
of funding from the Department of Science and Technology (DST) to 
hasten the development of the field and was managed by the National 
Research Foundation (NRF). O’Connor and Shumate (2014), despite using 
the US as the context, provide a model of operation involving business 
entities and NGOs collaborating in the areas of development, generally 
understood as corporate social responsibility in business circles. In the 
proposed new developmental model, NGOs and corporations collaborate 
in a manner in which NGOs, corporates and the government work in a 
symbiotic manner, triggering acceptance by the populations. Such 
collaborations are believed to (O’Connor & Shumate 2014): 

[...] meet the cultural expectations [recipients of the development] have about 


the role of business in society, while advancing social issue awareness and 
enhancing organizational legitimacy. (p. 106) 


In other words, if one alludes to universities, for example, as institutions or 
corporates entering into partnerships with developmental organisations, 
such as is the case between the Midlands State University (Zimbabwe) and 
UNICEF-Zimbabwe (who signed a memorandum of understanding in 
2022), it follows that the research and community services the university 
renders lay the foundation for evaluation and sustainable development. 
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By extension, reference to IKSs and their unavoidable use by developmental 
institutions such as NGOs reveals the resilience of IKSs (Chilisa 2015; Chilisa 
& Mertens 2021; Mbava & Chapman 2020). In the face of the shifting priorities 
of development or its failures, IKSs have not changed their epistemological 
and institutional underpinnings (Fernando 2003). Developmental institutions 
have come to realise that their calls for equity, especially pinned around the 
need to promote and accord respect to ‘human rights’, reflects the undeniable 
need to shift and revisit development as IKSs place indigenous people who 
have been calling for the protection of their environments from the onslaught 
of development (Fernando 2003). The MAE approach is therefore the 
alternative (Chilisa 2015; Chilisa & Mertens 2021) to guaranteeing equity, 
given how it has contributed significant knowledge and making indigenous 
knowledge ‘a framework of ideas, guidelines and institutional foundation 
that can offer entry points into alternative ways of thinking about sustainable 
development’ (Chilisa & Mertens 2021, p. 55). 


Studies by Bjørnholt and Larsen (2014), House (2015) and Hujo (2021) 
have demonstrated that inequality in evaluation has resulted from 
developmental approaches and methods imposed at the micro level. Taking 
a macro level view, the result has been perennial inequities and inequalities 
perpetuated by global and highly connected international institutions (see 
Hancorck 1989). Against such realities, it follows, therefore, that IKSs have 
become important sources and alternatives for the Global South to overcome 
the limitations and failures of development programmes largely driven and 
directed by the Global North. It is a cultural-based strategy to ‘guard’ the 
self-imposed ‘guards’ of development. Evaluations are often ‘political’, hence 
providing grounding for inequalities. But MAE, as an alternative paradigm, 
has political power drawn from IKSs while keeping in check the ever- 
expanding vicious capitalism. The global political economy invites MAE to 
re-orient aspects of equality for the common good. In Ghana, the aforesaid 
is noted around the mother-queen ‘MKQMA’ concept or institution. Given 
the prevalence of HIV and AIDS cases in Ghana, Drah (2014, p. 18) reported 
on the successes achieved by the MKQMA, a ‘traditional political system’ 
turned into an NGO, wherein ‘the queen mothers could directly raise support 
for orphans, thereby assuming the role of chiefs as providers of resources for 
childcare’. The episteme involved herein is linked to the mothering concept 
in African tradition wherein childcare, particularly in the areas of bringing 
equity among all children in Ghana and globally in the face of the HIV and 
AIDS pandemic; the MKQMA rose up as an intervention. International 
organisations such as UNICEF would only be required to render support in 
technical and economic means. However, as Drah (2014) noted: 

The MKQMA has been critical in improving the lives of women and children and 


developing a model for supporting orphans, but there is a tendency for NGOs 
concerned with HIV to claim more credit than they actually deserve. (p. 18) 
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This reveals how NGOs, because of exclusionary evaluation practices, do 
not recognise IKSs and are quick to erroneously conclude that any 
successful intervention to a developmental issue in Africa is based on the 
epistemologies of the Global North. 


In another show of individualism, hence inequity, by NGOs working in 
the African region, Ferrari (2011) indicated that the fight against HIV and 
AIDS reportedly showed that non-Roman Catholic NGOs shy away from 
collaborating with Roman Catholic-based NGOs deemed advantaged given 
their grip on health care, infrastructure and other resources. Part of the 
reason is the incompatibility of the NGOs (Ferrari 2011), for example, the 
Roman Catholic stance on condom use. NGOs in this regard have shared 
challenges, but they fail to collaborate on the solutions. The result is a 
systematic fragmentation of the health care systems from which NGOs 
succeed at ‘causing permanent harm to public systems of care’ (Ferrari 2011, 
p. 87). Hence, MAE is strategically positioned as both a strategy and a tool 
for harmonising counterproductive work while providing a very important 
bridge to accord collaborations some footing that will profit the organisations. 


In the Igbide, Uzere and Olomoro communities in the Delta state of 
Nigeria, Ebohuoma (2020) indicated that almost 90% of the households are 
directly or indirectly involved in agriculture, and all farmers rely heavily on 
IKSs for their agricultural practices, especially regarding meteorological 
issues. This means that evaluations ought to draw lessons from MAE, which 
advances frameworks that result in the integration of orthodox systems 
with IKS. For instance, Holma and Kontinen (2011) reported on reproductive 
health care efforts, foreseeing the successful HIV prevention intervention in 
South Africa involving the embedment of sexual education with existing 
practices during the tradition of girls’ initiation festivities. The festivities 
are part of a traditional practice conducted in order to educate girls and 
encourage avoiding pregnancies. 


This way, equity in evaluation is guaranteed, whether it is in HIV 
prevention and care, childcare, seasonal climate forecasts or food security. 
The volunteerism concept, which most NGOs have since formalised, has 
always been part of the livelihoods of African people (Kabonga 2020). The 
tradition of the harambe [let us get together] in Kenya and the zunde 
ramambo [chief’s granary] in Zimbabwe are cases in point. Premised on 
the ubuntu epistemology, the Global South’s take on volunteerism 
emphasises ‘dedicating oneself to a certain cause without expectation of 
payment’ (Kabonga 2020, p. 3), wherein the drive is to foresee the 
community benefiting. But today, in Africa, volunteerism has reportedly 
been re-structured to assume capitalist undertones or formalised to serve 
the purposes of NGO work, among them M&E officers who now rely on 
volunteerism to conduct data-collection, for instance. 
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Thus, MAE comes in to provide equity in evaluation by rectifying 
misrepresentations, such as the case cited by Drah (2014). Allowing 
international organisations to continue to dictate the pace of development 
and deliberately vie for credit that is not due to them (Drah 2014) has 
detrimental effects. Evaluation reports may reveal that, in the Ghanaian 
case, a successful developmental or interventional programme was 
exported to other African countries where the mother-queen concept is 
not part of the indigene, recalling that the African region itself is not 
homogenous. As a result, there is a manifestation of inequality fostered 
by the imposition of programmes purported to have been crafted in 
Africa and expected, as such, to be applicable anywhere in Africa with 
minimal modifications. 


The other advantage offered by MAE, as an option, is that realistic 
evaluation (Holma & Kontinen 2011) is ensured. When knowledge is 
produced through evaluation reports by NGOs, it must be realistic and 
representative, as failure to meet this basic requirement makes the 
knowledge a perpetrator of inequality in evaluation itself. In order to 
promote equity in evaluation, Holma and Kontinen (2011) noted that 
developmental institutions have flexed up their evaluation models to have 
them connect to ‘learning’ for the simple reason that NGO work must 
consider the values of the targeted people. In this regard, MAE provides 
this symbiotic and pragmatic connection to evaluation and learning. 


Thus, once MAE, as an option, is adopted fully, even the knowledge 
distribution (Holma & Kontinen 2011) mechanisms change. Efforts in 
South Africa towards operationalising MAE have witnessed the hosting 
of the MAE hackathon by the South African MAE in collaboration with 
other key players in the evaluation-scape in September 2021. There has 
been development of the Zenda Ofir blog site, the CLEAR-AA paper 
series and the African Evaluation Journal driving calls for papers in 
search and promotion of research data around MAE. These activities, 
among others, and institutional work are instances of operationalising 
IKS knowledge transfer. 


As far back as 1998, the Partnership for Information and Communication 
Technology for Africa (PICTA) and the World Bank agreed to lead an 
indigenous knowledge for development initiative to help stimulate 
recognition, utilisation and exchange of indigenous knowledge in the 
development process. The partners of the Initiative are CIRAN/Nuffic, 
CISDA, ECA, IDRC, ITU, SANGONet, UNDP, UNESCO, WHO and the World 
Intellectual Property Organization (WIPO), with the World Bank being the 
lead partner. This is an important move to give IKSs agency as well as 
address evaluation dependency issues by the Global South on the Global 
North. It remains to be seen whether leading institutions such as the 
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World Bank will reform themselves out of ‘power’ to accord inequity and 
inequality a chance at a macro level in order for evaluation practices at the 
micro level to function sustainably. 


E Conclusion 


There is very little information regarding the governance and operational 
challenges of NGOs and other developmental agencies that are internal 
and a site for inequity or inequality for professionals working with and 
for such institutions. Lack of research evidence does not entail that forms 
of inequalities and inequities are absent, because they exist and have 
operational effects on the institutions. What is noted are ‘outbursts’ 
sometimes mentioned in passing during workshops or conferences on M&E 
with very little documentation to help all professionals operate in various 
institutions. The notion to ‘always place the people who are most impacted 
at the centre of conversations which seek to find solutions to problems 
affecting them’ has not been well practised within development institutions. 
Take, for instance, formulations of the SDGs’ scholarship claims, regarding 
which African countries were not fully consulted. Ironically, these are the 
recipients of the global orders that they need to incorporate into their day- 
to-day activities. 


The scholarship reviewed in this chapter reflected on the matrix between 
the Global North epistemes that ignore or override the Global South 
epistemes, from which the take is that this is deliberately done to create 
inequity and inequality in development. Such a situation, as discussed in 
this chapter, further impacts, at the micro level, notions of M&E, as these 
are also knowledge-based and, as such, equally affected for any M&E 
practices in Africa. 


The chapter noted that the field of M&E has been growing, particularly 
in Africa, where universities in countries such as South Africa, Uganda, 
Kenya and Zambia, among others, are training individuals who then assume 
positions in government and profit-making and non-profit-making 
institutions in a bid to operationalise M&E. However, a lack of a sound 
foundations in MAE hampers such efforts. 


This area remains a concern despite the dearth of literature around 
evaluating the international institutions in as far as how they perpetuate 
inequities and inequalities contemporarily. Perhaps, more important than 
anything else is further research on who is evaluating the international 
institutions, particularly ‘if they practice what they preach’; hence, ‘who 
guards the guards when the guards go on strike’. Institutional change in 
shifting the thinking about inequalities and inequities will not be effective 
unless there is a strong and deliberate focus in which these institutions 
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become agents of creating equality and equity among countries and 
institutions, not only at individual levels. 


There is no doubt that international organisations play an important 
role in eliminating inequities and inequalities, but they must be aware of 
their impact in order to prevent unintentionally reinforcing inequalities 
and inequities. Some recommendations are discussed in the following 
paragraphs. 


Firstly, it was noted that there is a dearth of research assessing 
international institutions and their contribution to inequality. Research on 
that is paramount to uncovering factors that contribute to inequality in the 
Global South. That research could trigger international development 
partners to self-introspect and change their modus operandi when it comes 
to providing aid to the Global South. This should help contextualise the 
modus operandi against the ‘one-size-fits-all’ approach. 


Secondly, it was noted that it is better to foster transparency and 
accountability than to sideline governments that oppress their citizenry. 
International organisations should promote and support equitable 
governance frameworks in the nations with which they collaborate. 
Advocating for transparent and responsible political institutions that 
actively engage with marginalised groups such as females, minorities and 
indigenous populations falls under this category. Making governments 
accountable to their constituents can aid in the reduction of corruption 
and the promotion of equitable resource distribution. 


Thirdly, universities should prioritise the utilisation of African ways of 
knowing and should advance their own epistemologies that are ontologically 
African-focused, as opposed to employing Global North episteme 
exclusively where there is a contextual and axiological difference. African 
universities should conduct robust research about IKSs and MAE and 
demonstrate growing evidence of what works, for whom, how and why by 
applying MAE and IKS theories, methods and paradigms. Such knowledge 
should be integrated into the M&E curriculum. Doing so closes the 
epistemological lacuna and addresses epistemological inequalities between 
the Global North and Global South (particularly in Africa). Such a deliberate 
action should be a requirement in M&E training (short courses, diplomas, 
etc.) or evaluation (Chilisa 2015; Chilisa & Mertens 2021). Where possible, 
Global North knowledge should be included as part of blending theories, 
methods and paradigms. 


Fourthly, there is a need for a paradigm shift where evaluation ought to 
involve both implementers of a programme and funders, as well as those 
for whom the programme was initiated. This will also entail reviewing 
structural aspects of M&E tools used in the field, including harnessing 
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orthodox M&E methodologies with indigenous knowledge-based ones for 
sustainable M&E practices. 


Fifthly, international organisations should aim to be inclusive and 
participatory in nature when designing programmes. The implementers or 
Global South countries should be consulted from the onset rather than 
simply being furnished with predetermined programme documents 
Cincluding theories of change and logical frameworks and reporting 
templates). Often, they are consulted to provide feedback on documents 
already developed - therefore, there is no genuine consultation of 
beneficiaries and implementers. Where consultation is carried out, it is a 
matter of ticking the box and sanitising the process. 


Sixthly, international institutions are required to limit and, where possible, 
remove conditionalities to financial aid and loans that require Global South 
or African countries being required to accept certain reforms and policies, 
as seen in the implementation of ESAP, without proper consideration of 
context. This has had detrimental effects on countries. 


Seventhly, balancing financial resource allocation among Global South 
countries is critical, as these are unevenly distributed and, therefore, 
development even among these countries is not uniform. 
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Unlocking climate change 
adaptation injustice through 
using intersectional analysis 
as an evaluation approach 


Steven Masvaure 

Centre for Learning on Evaluation and Results, Anglophone Africa (CLEAR-AA), 
Faculty of Commerce, Law and Management, University of the Witwatersrand, 
Johannesburg, South Africa 


E introduction 


| begin with a real-life story of 35-year-old Lydia, a married female living 
in the rural southern part of Zimbabwe. Her community is in an area 
characterised by perennial drought and low rainfall. Lydia’s community was 
moved by white colonial settlers in the early 1900s from the wet and fertile 
highlands of central Zimbabwe to the southern part of Zimbabwe. Attempts 
by her community to go back to their ancestral land during the colonial era 
were thwarted by the colonial government through violence and various 
land dispossession legislations. During independence (post-colonial era), 
the new government denied or ignored the community’s call to return to 
their ancestral land. Within the community, there is resignation to the fact 
that the community will never go back to its ancestral land. Her story and 
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that of her household are similar to her fellow community members’ 
experiences in the communities located in the southern parts of Zimbabwe. 
Lydia has been married to Ronald, a 43-year-old male who used to be 
employed as a sugar cane cutter in sugar cane plantations located in the 
lowveld of Zimbabwe, for eighteen years. Ronald was retrenched five years 
prior to the writing of this chapter and is now a full-time peasant farmer. 


Lydia has five children who are attending local primary and secondary 
schools. Within her household, Lydia is expected to play culturally assigned 
roles such as taking care of the children, cooking, cleaning, farming and 
other roles assigned to female household members. Her farming duties are 
centred around producing food crops that contribute to household food, 
such as vegetables, groundnuts, Bambara groundnuts, beans, sweet 
potatoes and other crops that are culturally regarded as a preserve of 
female members of the household. Her husband’s farming activities are 
also defined by cultural expectations, and he is responsible for staple crops 
such as maize and other small grains that are used for traditional and 
religious ceremonies. Although there is a cultural allocation of crops to 
household members (especially the husband and wife), labour is shared 
across all household members. However, crop ownership rests with specific 
household members. Overall, the household land is owned by Ronald by 
virtue of him being the head of the household. 


Lydia’s household’s farming activities are conducted on a small piece of 
land that is communally owned. Every year, she has to negotiate with her 
husband for more land to plant crops that contribute to the household 
food basket. However, the response from her husband is that their land 
cannot expand; hence, she has to make do with whatever is available. Her 
land predicament is exacerbated by the continuous degradation of the 
land that is resulting in a progressive decline in yields. 


In Lydia’s community, several non-governmental organisations (NGOs) 
are supporting the households within the community to cope with climate 
change. Some of the organisations are focusing on new climate-smart 
agriculture methods, while others focus on introducing cash crops to 
improve household incomes. In addition, other NGOs are supporting 
households with food aid during years of drought. These organisations 
have been working in these communities since time immemorial. What 
changes in their operations is only the intensity of their interventions; 
during seasons of better yields, they reduce their intensity while maintaining 
a presence focusing on the poorest households. 


Although Lydia appreciates that the various programmes supporting 
households brought some positive benefits, there are several challenges 
that they caused within the households and the community at large. Firstly, 
she views programmes such as those that support cash crops as mainly 
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benefiting the male members of the households as cash crops culturally fall 
within the male domain. The male member or husband controls the income 
received when crops are sold and the land for cultivating cash crops is 
taken from land preserved for household food and vegetables. Secondly, 
new ways of communal farming that are being promoted by the 
organisations in the community reduce the size of land available for 
agriculture through the reduction in fertile stream bank cultivation, which 
in turn reduces the yield. Also, the organisations are advocating for a 
reduction in livestock because of the purported overgrazing - this affects 
the household store of wealth (rural households store wealth in the form of 
livestock) and food security, as the livestock produce milk and meat for 
household consumption. Thirdly, although food aid assists households in 
coping with food insecurity, it has created a dependence on these 
organisations and households are no longer exploring other sustainable 
opportunities to address food insecurity. Finally, organisations that are 
advocating for the preservation of the environment are also causing 
hardship within the community as traditional leaders are now policing 
access to firewood, a major energy source for the community. Although 
Lydia’s household understands the intentions of those offering the 
community help, she wishes that offering help could aid in understanding 
the predicament and vicious cycle that the households in her community 
experience. 


Central to this chapter is the examination of the various forms of 
inequalities that Lydia and her community face when they try to adapt to 
climate change. Although Lydia’s household and community are fully aware 
of their precarious condition and the negative consequences of some of 
the programmes that seek to help them, they have adopted a ‘quiet 
acceptance’ approach because they feel that they do not have the power 
to deal with the course of climate change adaptation. The households have 
internalised the climate adaptation injustice and seem to prefer the 
mistreatment and perpetuation of their own oppression. The household’s 
and the community’s preference for climate change adaptation has been 
subverted. The circumstances in which Lydia and the community find 
themselves have led to adaptive preference formation - adaptive 
preferences refer to instances where deprivations or inequalities are not 
seen as such by the individuals and households who experience them. This 
situation is best described by Freire (2018) when he said: 

However, the oppressed, who have adapted to the structure of domination in 

which they are immersed, and have become resigned to it, are inhibited from 


waging the struggle for freedom so long as they feel incapable of running the 
risk it requires. (p. 37) 


This chapter will demonstrate how the use of the intersectional approach 
in evaluation contributes to a better understanding of the climate injustice 
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problem. Furthermore, this chapter will explore how the various forms of 
inequalities influence climate change adaptation decision-making. 


E Research methods 


This chapter is based on personal experiences and experiences of the 
communities in the southern parts of Zimbabwe. My personal experiences 
of interacting and living among southern Zimbabwean communities are 
coupled with accumulated tacit knowledge through professional evaluation 
experience, discussions with fellow evaluators and my work in the evaluation 
capacity development space in Africa. The methodological approach 
adopted in this is termed autoethnography, which is defined as an approach 
to research and writing that seeks to describe and systematically analyse 
(graphy) personal experience (auto) to understand cultural experience 
(Ellis, Adams & Bochner 2011). My objective in this chapter is not only to 
narrate personal experiences but also to expand the understanding of the 
social realities of communities that are facing climate risk and are trying to 
adapt. In this chapter, | am going beyond mere narration of my experiences 
to engage in cultural analysis and interpretation - such an approach aligns 
with Chang’s (2016) definition of autoethnography (Chang 2016). This 
chapter begins by presenting the real-life story of a household in the 
community. In subsequent sections of the chapter, a discussion follows on 
how the intersectional approach as a tool can be used to understand the 
various forms of inequalities affecting individual and household responses 
to climate change risk. The final section of the chapter discusses key 
strategies for evaluators to use when evaluating programmes that are 
being implemented in such communities. 


E Intersectional framework and Lydia’s 
predicament 


Key to Lydia and her community’s story is the concept of ‘quiet acceptance’. 
| am characterising it as quiet acceptance because there is no resistance 
from the individual and community, mainly because the community does 
not have a genuine voice. The community voice has been throttled by 
unequal power relations between those who are trying to help the 
community adapt and the community itself. Quiet acceptance can be linked 
to literature from the Global South that focuses on internalised oppression 
and adaptive preferences. ‘Internalized oppression is the incorporation and 
acceptance by individuals within an oppressed group of the prejudices 
against them within the dominant society’ (Pheterson 1986, cited in 
Williams 2012, p. 32; cf. David 2013). In Lydia’s case, internalised oppression 
has left her feeling resigned, inferior, powerless and at the same time 
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grateful to the organisations that are helping her household and community 
to survive. Such a mixture of emotions and perceptions requires that 
evaluators interrogate this insidious consequence of oppression that is 
inadvertently perpetuating the various forms of inequality. 


Of primary concern to overcome the oppression that Lydia and her 
community are facing is the fact that they have become accustomed to 
their circumstances and even come to prefer them - adaptive preferences. 
Adaptive preferences refer to instances where deprivations and inequalities 
are not seen as such by the individuals who experience them (Colburn 
2011). Nussbaum argued that such ‘deformed’ preferences shaped by 
inequality do not lead to better well-being - what it does is it makes it 
difficult to critique unjust practices and institutions (Nussbaum 2003, 
2009). This inability to critique unjust practices and institutions has 
encroached into the evaluation space where evaluators validate the status 
quo without necessarily being critical (Paz-Ybarnegaray & Douthwaite 
2017). In the context of deformed preferences, there is a likelihood of 
evaluations revealing the level of satisfaction with the climate change 
adaptation interventions that are being implemented in those communities, 
creating what Sen called a happy slave (Sen 2001). Beyond these external 
inequalities, other intra-household inequalities affect females, and these 
are culturally regarded as settled and have adaptive preferences such as 
curtailed labour market access, denial of access to education, unequal 
childcare, unequal decision-making power and unequal access to household 
resources. These are common and persistent to Lydia, and she has accepted 
them. Caution needs to be exercised by the evaluators when interpreting 
these subjective aspects that determine the success of interventions that 
seek to address climate change risks. This chapter argues that evaluators 
need to employ methodologies that are fit for purpose and have the ability 
to reveal the internalised oppression and adaptive preferences that 
perpetuate inequality, especially in the realm of climate change adaptation. 
In this chapter, intersectional framework/analysis is a central and relevant 
tool for evaluators as they help reveal the underlying causes of oppression, 
internalised oppression and adaptive preferences. Intersectionality is 
defined as ‘a metaphor for understanding the ways that multiple forms of 
inequality or disadvantage sometimes compound themselves and create 
obstacles that often are not understood among conventional ways of 
thinking’ (Crenshaw 1989, p. 140). Furthermore, intersectionality ‘aims to 
shed light on how multiple dimensions and systems of inequality interact 
with one another and create distinct experiences and outcomes’ (Corus & 
Saatcioglu 2015; Scottish Government n.d., pp. 13-14). The intersectional 
approach has its origin in feminist critical analysis. Through using 
intersectionality, evaluators will be able to examine how forms of oppression 
and deprivation shape people’s experiences. 
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In this chapter, there are three key observations of intersectionality to 
which | will pay attention. To begin with, intersectionality recognises ‘that 
people are shaped by their simultaneous membership in multiple 
interconnected social categories’ (Wijeyesinghe & Jones 2014). Furthermore, 
intersectionality dictates that ‘the interaction between multiple social 
categories occurs within a context of connected systems and structures of 
power (e.g. laws, policies, and governments) (Shannon et al. 2022; Scottish 
Government n.d., p. 12). In addition, intersectionality highlights that 
structural inequalities are a result of the interaction between social aspects, 
power relations and contexts. Hence, an individual’s experiences of 
inequality are chronic and create distinctive lived experiences. Finally, the 
chapter establishes that: 


1. inequality is intersectional 

2. inequality is always specific to an individual, location and social group, 
and awareness of these specifics is essential to the design of interventions 
and programmes intended to address inequality 

3. despite differences in the way inequality is experienced by individual, 
different groups and in different places, there are striking commonalities 
in the experience of inequality in very different countries, communities 
and globally. 


Poor people’s climate change adaptation options are characterised by 
powerlessness and voicelessness, which limit their choices and define the 
quality of their adaptation options. 


The intersectional approach does not give a higher status to a specific 
inequality or experience of discrimination (Evans 2019). However, the 
international development space and the interventions that are implemented 
show that there are specific inequalities that are prioritised depending on 
the focus of the organisation implementing them. The international 
development community tends to assume an unquestioned similarity of 
inequalities, leading to a failure to address the structural level of inequality 
and also fuelling the political competition between inequalities (Verloo 
2006), thereby doing a disservice to the intended beneficiaries such as 
Lydia. This paper argues that the ‘one-size-fits-all’ approach to addressing 
multiple dimensions of inequality is based on an incorrect assumption of 
the similitude of the social categories connected to inequalities and of the 
mechanisms and processes that constitute them. By focusing on similarities, 
one ignores the intersectional character and dynamics of inequalities. 
Literature has shown that addressing one inequality does not mean that 
the other inequalities are also addressed. In some cases, addressing one 
inequality will provide relief to the other inequalities (Seedall, Holtrop & 
Parra-Cardona 2014), but in other situations, this will aggravate the impact 
of the other inequalities. Intersectionality dictates to us that Lydia’s identity 


94 


Chapter 5 


is acomplex amalgamation of different categories of inequality that shape 
her life experiences. The next section re-examines Lydia’s situation 
intersectional framework. 


E Lydia’s multi-dimension of inequality 


Let us go back to re-examining Lydia’s situation. Lydia and her community 
are facing several dimensions of inequality, and these dimensions of 
inequality shape her and the community’s experiences and how they 
respond to climate change risk. To begin with, Lydia is affected by structural 
inequality - structural inequality involves uneven power relations between 
groups of people and is often difficult to address because it is not recognised 
as unjust (Pease 2021). Lydia and her community face structural inequalities 
that emanate from the colonial era when their communities were displaced 
from fertile wetlands - this is well-documented in Zimbabwe's colonial 
history. Colonial history has shown that the dispossession was wanton, 
violent, racist and spearheaded by the colonial interests of the countries 
from the Global North. The general perspective of Lydia’s community is 
that the current purported climate change risk and vulnerabilities that the 
international development community are focusing on are not necessarily 
climate change induced but structurally induced. The consensus is that if 
the community was not dispossessed of its fertile land, they would not be 
forced to practice subsistence agriculture on infertile and dry land. The 
general feeling is that post-independence in Zimbabwe, the ‘system’ or 
government did not put in an effort to address this unjust atrocity and 
return them to their land. Although there was land reform 20 years after 
independence, it was political and elitist to the extent that vulnerable 
communities were left out in favour of the political elite (Zamchiya 2011). 


What is also unknown to Lydia and her community is that their non- 
return to their ancestral land was not the decision of the post-independence 
government only but was also influenced by global players that included 
the former colonisers, multilateral institutions and the broad international 
development community that lobbied for the preservation of property 
rights, that is, preserving the land ownership rights of the former colonisers 
regardless of how they acquired land (Césaire 2019). Given the international 
balance of power (inequality of powers between the Global North and 
Global South), this issue was never resolved till the country went through a 
chaotic land reform programme that was politically driven and neglected 
the most vulnerable communities. Lydia and her community have resigned 
themselves to the fact that they cannot be returned to the land of their 
forefathers, and their adaptive preferences have been limited to the options 
that are being offered, largely by the international community. What is 
important to note is that Lydia and her community’s adaptive preferences 
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no longer reflect their aspirations of being relocated to their forefather’s 
land. However, deep down, they believe that if they are relocated, they will 
be able to grow their food and not depend on handouts. 


On the other hand, Lydia is also facing other inequalities that are based 
on her being a female. This form of inequality has been termed gender 
inequality. Gender inequality is discrimination based on sex or gender, 
causing one sex or gender to be routinely privileged or prioritised over 
another (Johnsson-Latham, 2007). Several components are affected by 
gender inequality, and these include labour market participation, paid work 
and unpaid work, income and intra-household sharing and intra-household 
self-employment and decision-making. In lydia’s case, it is important to 
highlight that climate adaptation decisions are made in the context of the 
household, which is defined by the cultural and social norms that influence 
the division of roles between Lydia and her husband. By virtue of Lydia being 
a female, she is expected to perform household tasks that are culturally 
assigned, like childcare and a range of other domestic chores or housework, 
while her husband is expected to participate in the labour market and be 
involved in activities that earn income for the household. However, he will 
have more decision-making power on how the income is used. 


Despite Lydia and her husband, Ronald, being in one household, there is 
some independence in some economic activities between herself and her 
husband. The division of labour also shows that when it comes to climate 
change adaptation, there is aneed for adaptation options that take cognisance 
of the household context and dynamics. Most programmes funded by the 
international community focus on climate change adaptation options that 
seek to address household food insecurity by focusing on adaptation strategies 
that increase the available household staple food. The implication is that the 
staple food, which is maize, is culturally assigned to the male member of 
the household, meaning that adaptation programmes focus more on male 
members of the household as compared to the female members who are 
involved in the cultivation of other crops such as vegetables, beans, etc. 
This creates inequality and perpetuates the ingrained gender inequality where 
more value and prioritisation of crops benefit the male members of the 
household. The key argument by those funding and designing climate change 
adaptation programmes for communities is that there are limited financial and 
resources for holistic programmes that address a range of inequalities that 
communities face. With that, | argue that although human beings are diverse 
and some of the diversities are left out because of the need for simplification 
of climate change adaptation options, there is a need to understand the risks 
of oversimplification of the programmes. 


Another compounding insidious gender inequality factor affecting Lydia 
is that the programmes focusing on helping households generate income 
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through the growing of cash crops are doing more harm to her. As stated 
earlier, Lydia feels that these programmes, instead of addressing the intra- 
household income disparities and inequities, are worsening it in several 
ways. Firstly, because of limited land, the land for cash crops is taken from 
non-staple food land, meaning it is Lydia’s land allocation that is affected, 
thereby reducing the land available for other crops that are vital to 
household nutrition, extra money for the children’s clothing and other 
needs. Furthermore, it limits Lydia’s ability to earn an income from excess 
crops, thereby affecting her independence and ability to address the 
inherent gender inequality. This supports the observation by Islam and 
Winkel (2017) that inequality increases the exposure of disadvantaged 
social groups to the ‘adverse effects of climate change’. In addition, given 
Lydia and her household’s exposure level, inequality increases their 
susceptibility to damages caused by climate hazards because the 
programmes that are seeking to address climate hazards and risks are not 
paying attention to the various inherent types of inequalities within 
households and communities. Furthermore, intra-household inequality 
decreases the household’s relative ability to cope with and recover from 
the climate risk and damage they suffer. In summary, the issue of cash 
crops means that males are taught and encouraged to grow cash crops 
such as cotton, while females continue to grow crops with less market 
value with minimum support, thereby perpetuating income disparities. 


Lydia’s situation also shows the conundrum that is faced by communities 
in the Global South when they try to address climate change through 
adaptation and mitigation. Although climate-smart agricultural practices 
are commendable, their implementation should be guided by the local 
context. Lydia’s household views climate-smart agriculture practices such 
as limiting the number of livestock they can keep, leaving land fallow for 
some years, banning stream bank cultivation and restrictive use of natural 
resources such as cutting down trees create a new challenge for her as a 
female member of the household. Although there are no questions about 
the nobility of these climate-smart interventions, they need to be viewed 
in the context of poor access to productive land and other household 
dynamics. At the individual level, the new climate-smart agriculture 
approaches that are being implemented and pushed by the international 
development community and government have limited Lydia’s access to 
the household energy source: firewood. She must now spend more time 
looking for firewood, given the fact that she is no longer allowed to cut 
down trees. At the household level, the overgrazing initiatives that limit 
the number of household livestock, such as cattle, have negatively 
affected her household’s store of wealth. Traditionally, domestic livestock 
has been used as a store of wealth and also contributes to the household 
workforce for farming and food security through the provision of milk, 
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meat and cash when sold. The community and households are aware of 
the implications of these interventions; however, they feel that they are 
disempowered to challenge them, as most of the organisations that are 
implementing these programmes work with government departments 
and local traditional leaders. Given the power dynamics within this 
community, the households and the community feel that they do not 
have a voice and cannot fully express how they feel about the design 
and implementation of these programmes that are aimed at helping them 
to adapt. 


E What should evaluators do to prevent 
perpetuating inequality? 


In this section, | will explore the nexus between evaluation, the intersectional 
approach and climate justice, that is, inequalities that are associated with 
the current climate change dilemma. The deliberate intention is to highlight 
the key current evaluation practices in climate change and how they help 
or do not help in understanding inequalities associated with climate change. 
| will also discuss how adopting an intersectional approach as an evaluation 
approach provides a better understanding of inequality and eventually 
assists the intervention designers and implementers in holistically delivering 
climate justice and addressing other forms of inequalities within households 
and communities. 


As | have fairly discussed the intersectional approach and climate justice, 
| will commence by giving a brief definition of what evaluation is. The 
evaluation field is littered with several definitions of evaluation. The 
American Evaluation Association defines evaluation as a systematic and 
intentional process of gathering and analysing data (quantitative and 
qualitative) to inform learning, decision-making and action (American 
Evaluation Association 2014). On the one hand, Scriven (1991) defines 
evaluation as referring to the process of determining the merit, worth or 
value of something or the product of that process (Scriven 1991). What is 
ingrained in these definitions is that evaluation is a tool that aids decision- 
making by making a judgement of the merit, worthiness and value of the 
intervention. What is imperative is to ask the question: Is the evaluation 
being used as a tool to address inherent inequalities, or is it just focused on 
the symptoms of inequality? Evaluation, in its truest sense, should provide 
evidence that informs how programmes are designed and implemented. 
Hence, the expectation is that it should play a critical role in bringing equity 
to society. | argue that, as evaluators, we should heed Freire’s words that 
we should not wash our hands of the conflict between the powerful (those 
funding and implementing climate change adaptation) and the powerless 
(communities trying to adapt) means to side with the powerful and not be 
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neutral (Freire 2018). | am now going to present some suggestions that 
evaluators can adopt to holistically address inequalities in climate change 
adaptation. 


Causal analysis of inequality 


| posit that evaluation interlinks with multitudes of social needs and must 
be responsive to diverse, intersecting identities and experiences. By doing 
so, evaluation will reveal the underlying causes of inequality and how 
inequalities within communities shape the lived experiences of the 
individuals and communities. Although there are scholars who have argued 
that evaluation should take a narrow peak at the intervention that is under 
scrutiny, if evaluators want to be of service to the Global South communities, 
it is prudent for the evaluators to broaden their scope and examine the root 
causes of the problems they are addressing, for example, in evaluating 
climate change adaptation interventions it is common practice for 
evaluators to have a narrow focus on the interventions and how they are 
contributing to a specific need in the community without necessarily 
interrogating the historical and the root causes of the problem within these 
communities. The key question that evaluators should ask is how the 
communities arrived at this point of vulnerability as compared to what the 
intervention achieved. Climate change programmes address the symptoms 
of the underlying powers that define inequality. 


My argument is that most of the development challenges in the Global 
South have their origin and are deeply ingrained in historical inequalities 
that programme funders, designers, implementers and evaluators fail to 
recognise and acknowledge. This approach is pervasive within the broad 
international development space. For example, it is common knowledge 
that the purported climate change challenges, such as the increasing 
poverty, food insecurity and climate risk within rural communities, have 
their roots not in current climate change but in violent and racist historical 
dispossession of land and other means of livelihoods during the colonial 
period. This dispossession was never addressed post-independence. Post- 
independence, there was a reset, or the slate was wiped clean without 
addressing the historical inequalities that disadvantaged communities. An 
evaluator using an intersectional approach rejects this approach to post- 
colonial development as it is an additive model of oppression that leaves 
the colonial systems that create power differentials unchanged (Hancock 
2007). | am arguing that evaluators need to ask the question: What are the 
root causes of the climate risk in these communities? Understanding the 
causes of inequality requires one to understand the society’s social fabric, 
how it is set up and who benefits from such a setup. A causal analysis of 
inequality points to structural and social relationships that determine who 
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gets what (Shaffer 2015). In evaluation, this has been missing; the focus has 
been on specific symptoms of inequality like poverty, food insecurity, etc. 
Thus, if the root causes of vulnerability are not considered, potential 
solutions might exacerbate rather than reduce existing injustices while 
leaving the challenges of climate change unaddressed (MacGregor 2010). 


Measuring inequality in evaluation 


The current approach to measuring inequality in evaluations is influenced 
by economic discipline measures such as the Gini coefficient, the Lorenz 
curve and other natural science quantitative methods - this approach 
aligned with Sen’s observation that inequality is judged by comparing 
some particular aspect of a person (such as income, wealth, happiness 
liberty or opportunities or rights or need fulfilment) with the same aspect 
of another person; in this case, the aspect is the ability of the communities 
to respond to climate risk (Sen 1995). Although these approaches are 
globally accepted, they have limitations. One such key limitation is that 
these two measures fail to capture the nuances and the context of 
households, and also, they emphasise the importance of income inequality 
at the expense of other types of inequalities; hence, these two measures 
will not allow the evaluators to understand Lydia’s context and the 
various forms of inequality that affect her and her household. The 
intersectional approach does not only focus on addressing a specific form 
of oppression but also on rooting out all forms of oppression. This is 
supported by the observation that climate change adaptation programmes 
reduce people to a single category at a time - they do not look at the 
intersectionality and are based on the assumption that social categories 
(i.e. race or ethnicity, gender, class, sexuality and ability) are constructed 
and dynamic (Cho, Crenshaw & McCall 2013; David 2013). Evaluators 
should take note that the effects of climate change are mediated through 
social, cultural and economic structures and processes, and the need for 
social analyses concerning the issue has become more recognised 
(Kaijser & Kronsell 2014). 


The current prominent approach in evaluating climate change adaptation 
interventions is to use the household as a unit of analysis. In addition to the 
household unit analysis, there has also been some foray into distinguishing 
whether a household is female-headed or male-headed - this is being done 
to capture the gender differences. | posit that an analysis of female-headed 
households does not work because it does not focus on intra-household 
vulnerabilities such as those espoused by Lydia. Evaluators need to 
differentiate intra-household vulnerabilities and how they play out even in 
male-headed households. The intra-household vulnerabilities determine the 
adaptive preferences and the adaptation actions that a household takes. 
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Inequalities are embedded and the current 
system of international development 
reproduces inequalities 


Evaluators need to understand that inequality is embedded in our social 
structure, and the search for a solution requires us to examine all aspects 
of our society (Atkinson 2014, p. 620). The current discourse on climate 
change adaptation inequality or justice focuses more on the inequalities 
across the Global North and Global South. Key debates and discussions 
focus on the causes of climate change and who is responsible for adaptation 
and mitigation, with very limited discussions and debates on in-country 
and intra-household inequalities and how they affect climate change 
adaptation (Islam & Winkel 2017). Social inequalities within the country, 
communities and households have not been at the forefront of climate 
adaptation decision-making. The current financing models focus more on 
funding climate change adaptation and mitigation in the Global South; 
however, the funds are not directed at social inequalities that perpetuate 
and increase exposure to climate risk. The non-dominance of social 
inequalities has meant that evaluations have not focused on understanding 
the embeddedness of inequality and how the current approach to climate 
change adaptation is aiding the reproduction of inequalities. Evaluations 
on climate change adaptation projects and programmes have focused on 
the impact of the programmes but have failed to examine social inequality 
in climate change adaptation. 


I am arguing that evaluation interconnects with myriad social needs and 
must be responsive to diverse, intersecting identities and experiences. 
Evaluations can play a critical role if they examine the intersecting factors 
and conditions by which inequalities (in adapting communities) are not 
only produced and reproduced but also actively resisted. An intersectional 
approach calls for a more complex approach to address the system that 
creates inequality rather than just the symptoms thereof (Hancock 2007). 


Evaluators should understand that the capacity 
to adapt is shaped by power relations 


Climate change adaptation refers to those crucial actions or plans that a 
country, community, household or individual will employ against a current 
or anticipated impact of climate change (Singh et al. 2022). These actions 
and plans are largely influenced by the adaptive capacities (‘capacity to 
learn, combine experience and knowledge, adjust responses in a pro- 
active way to changing external drivers and internal processes, and 
continue operating’ [Berkes 2003]), absorptive capacities (includes all 
the various risk management strategies by which individuals and 
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households moderate, or cope with the impacts of shocks on their 
livelihoods and basic needs [Smith & Frankenberger 2018]) and the 
transformative capacities: 
[T]he capacity to create an enabling environment through investment in good 
governance, infrastructure, formal and informal social protection mechanisms, 


basic service delivery and policies/regulations that constitute the necessary 
conditions for systemic change. (n.p.) 


When evaluators apply an intersectional lens to these different types of 
capacities, they will realise that the capacity to adapt and respond to 
change is shaped by power relations determining access to resources, 
information and the availability of options and choices (Tschakert & 
Machado 2012; Djoudi et al. 2016). This is critical to the understanding of 
the key causes of inequality and how individuals, households and 
communities respond to the intervention. Evaluators should also focus on 
the ability of climate change interventions to trigger the change in these 
social inequalities. 


Evaluators need to understand that socially and 
economically disadvantaged and marginalised 
people are disproportionally affected by climate 
change 


Literature on climate change vulnerabilities has shown that socially and 
economically disadvantaged and marginalised peopleare disproportionally 
affected by climate change (UNFCC 2018). IPCC (2014) also noted that 
socially and geographically disadvantaged people - including people 
facing discrimination based on gender, age, race, class, caste, indigeneity 
and disability - are particularly affected negatively by climate hazards. 
However, what is missing from the literature is how an evaluator unravels 
these vulnerabilities that are caused by inequality. When an intersectional 
approach is adopted in this instance, it dictates the need to understand 
the effect of the programme, and there is a need to grapple with the 
social and economic inequalities and show how they are contributing to 
inequalities. An evaluator needs to understand the many types of 
inequalities - inequalities based on demographic characteristics, such as 
gender, race, ethnicity, religion, age, assets and income. Care should be 
taken to not just focus on a few of these. In addition to these inequalities, 
evaluators also need to examine inequalities regarding public decision- 
making (political power) and access to public resources, such as publicly 
financed health, education, housing, financing and other services. These 
inequalities shape how individuals, households and communities respond 
to climate change. 
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Evaluators must ask the right questions 


Finally, the Intersectional approach requires that evaluators adopt evaluative 
thinking, which is defined (Buckley et al. 2015): 
[...] as a cognitive process in the context of evaluation, motivated by an attitude 
of inquisitiveness and a belief in the value of the evidence, that involves skills 
such as identifying assumptions, posing thoughtful questions, pursuing deeper 
understanding through reflection and perspective-taking and making informed 
decisions in preparation for action. (p. 2) 


When an evaluator adopts an evaluation thinking lens, then the following 
questions should be asked by the evaluator when evaluating climate 
change adaptation programmes: What inequities exist concerning the 
climate change ‘problem’? Where and how can interventions be made to 
improve the problem? What are feasible short-, medium- and long-term 
solutions to the problem? How is the intervention contributing to the 
reduction of inequities? What is the evidence that shows if inequities 
have been reduced? In addition, evaluative thinking needs to be applied 
to be asked from the programme design stage, not just as part of the 
evaluation design. 


E Conclusion 


The thrust of this chapter is that there are multiple dimensions of inequality 
that affect communities’ adaptation to climate change. Evaluators need 
to employ appropriate approaches that address these multiple dimensions 
of inequalities. One of these approaches is the intersectional approach, 
which is critical to understanding how communities adapt amidst a 
plethora of inequalities. It is also critical to understand that, although 
understanding how the project has performed is important, it is also 
critical for evaluators to complement that understanding with evidence 
on how the communities become vulnerable in the first place and how the 
programme has affected the multiple varieties of inequalities. Furthermore, 
evaluations need to come up with approaches for measuring the various 
types of inequalities and how they affect climate change adaptation 
actions. 
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E introduction 


The climate adaptation agenda in food systems 
discourses: Unpacking the history of the problem, 
politics and policy dilemmas 


This chapter asks: ‘Whose agenda is it?’. It seeks to uncover how climate 
equity and justice can be reframed in the evaluation of agri-food systems 
discourses. But first, | will begin by briefly unpacking the current history 
and agenda. There is no better place to start than at the beginning. Often, 
these discourses inquire about the influence of the colonial era on agri- 
food systems and development, if at all. Stories of injustice and equity are 
understood differently depending on how the story is told. So, | begin with 
a brief account of the pre-colonial emergence of farming families and why 
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agriculture began in Africa with the history of the hunter-gatherer culture 
more than 10,000 years ago (Niemeijer 1996). 


Hunter-gatherers transitioned from their nomadic lifestyles to being 
farmers - they transitioned as groups increased in number from finding 
food sources primarily from hunting and gathering to clearing small areas 
of land, cultivating indigenous crops from seeds of sorghum and millet, 
among other crops, and mostly raising domesticated wild animals to serve 
their human needs. It not only changed how Africans obtained food but 
also how farming communities lived. Food was produced locally with less 
land per person required for survival on abundant land. African farmers 
residing in rural villages grew a varied range of indigenous food crops, 
timed planting to accommodate labour demands during harvesting and 
managed risk in a variety of ways. These systems were non-equilibrium 
(not static), developed and adapted over time, but persistent systems 
nonetheless. Agricultural systems were internally driven and based on local 
needs and surplus produce was used for trade opportunities to further 
local development (Bjorniund, Bjornlund & Van Rooyen 2022). Subsequently, 
this changed how people experienced life as they could expect to stay in 
one place all year round, establish villages and develop knowledge of land 
use management as well as landscapes to meet their needs. The question 
answered here is not when and where agriculture began - there are many 
origins, in fact, and there are countless theories - but why it began. 


E Colonial era - paradigm shift 


A paradigm shift took place from farming families to institutionalised 
export production as the dominant production system during the colonial 
era from the mid-1800s to the early-1900s. By 1914, about 90% of the 
continent’s territory was under the control of colonisers (Bjornlund, 
Bjornlund & Van Rooyen 2022). But why were European colonisers 
interested in Africa? The shift was based on the agenda set by Global North 
colonisers tracing back to the Berlin Conference of 1884-1885s. Colonial 
governments were pressured by European businessmen and corporations 
to increase investment and the development of policies to advance raw 
material extraction and intensification of mono-crop cultivation in colonies. 
This was done under the pretext of civilising the ‘dark continent’. There was 
no interest among Europeans for agricultural diversification or investment 
in indigenous commodities. Figure 6.1 depicts the impact of colonialisation 
on food security. 


For the purpose of this chapter, three results are highlighted: Firstly, 
these events were followed by colonisers’ displacing and dispossessing 
native inhabitants of their land. Resistance wars ensued between African 
indigenous tribes and colonisers, with the former fighting to protect land 
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Source: Bjornlund et al. (2022, p. 851). 
FIGURE 6.1: Impact of colonialism on food security. 


and themselves against displacement. Inevitably, colonisers stole the land 
and gained control over the agricultural and economic systems, with 
Europeans controlling the majority of the cultivable land. Secondly, many 
indigenous crops suffered as a result of this colonialisation process as 
farmers had no choice but to abandon their indigenous food crops. As a 
result, severe food shortages occurred, and agricultural production 
methods were redirected. Thirdly, during this time, the Germans, French 
and British set up research centres to ‘teach’ the most effective industrial 
high-input agricultural practices promoted as improved ways of crop 
cultivation. European industrial seed companies mass-produced, 


107 


‘Whose agenda is it?’ Reframing equity and justice in the evaluation 


documented practices, published, translated into African languages and 
distributed them to farmers for adoption. These colonial impacts are 
evident today across the African continent. 


E Post-colonial era - impact on agri-food 
systems 


During the post-colonial era, food systems in Africa remained fragmented 
and reliant on export-driven production systems, leaving subsistence 
smallholder farmers destitute and reliant on high-input agriculture and 
foreign seed varieties. Consequentially, local food systems are inherently 
off-track and most African communities and smallholders are in need of 
immediate inputs and food assistance. At the same time, the change in 
food systems also exacerbated the current food crisis; Africans are no 
longer dependent on traditional food sources but rather on Western food 
and seed varieties. 


In 2020, more than one in every five African suffered from hunger, more 
than doubling the proportion of hungry individuals in any other region, and 
over 282 million Africans are undernourished (World Bank 2022). At least 
28 million of these individuals have suffered from severe food insecurity in 
the last two years as a result of numerous shocks, with the most vulnerable 
suffering the most (Muleta 2022). More than 27 million people in West 
Africa alone required immediate food assistance in 2021 (Obayelu et al. 
2021). Food insecurity is on the rise in East Africa, where 7 million people 
are starving, and 33.8 million people are fighting for food (Food Security 
Information Network [FSIN] 2020). In southern Africa, over 27.4 million 
people have been estimated to be food insecure over the last two years 
(World Bank 2022). Despite this, African post-independence states 
continue to support high-input-led production systems and contemporary 
policy dilemmas such as foreign investor land acquisitions for mining 
investments, industrial developments and agricultural production (Nolte & 
Ledermann 2023) consistent with foreign interests based on neo-colonial 
agendas. 


Emergence of the climate agenda in smallholder 
agri-food systems 


The rise of global climate discourses in the 1950s ‘Anthropocene era’ 
debates, influenced by Global North scientists, was coined as the start of 
the devastating impact of human activities on the ecological system 
(Rafferty 2020; Whyte 2018; Yusoff 2018) and the birth of a new ideology 
(i.e. a set of ideas and framework), which spurred the need for climate 
adaptation and mitigation as well as the need for transformation of global 


108 


Chapter 6 


agri-food systems. The recognition and dominance of the Anthropocene 
era as the baseline placed on the climate crisis sparked outrage in the 
Global South because this frame fails to recognise the historical actions of 
a minority of Global North colonialists, capitalists and patriarchs on social 
and ecological systems (Whyte 2018; Yusoff 2018). The baseline of 1950 
negates the history of people who have been exploited for centuries by 
imperialist institutions, structures and systems. 


In fact, it was not until the late 1990s, with the rise of green politics, that 
interest in smallholder agriculture in Africa became a focal point of 
international development (Aliber & Hall 2012; Havnevik et al. 2007; Whyte 
2018). In part, Green Revolution land use policies’ promotion of high- 
yielding varieties of major cereal crops was justified by the narrow narratives 
of smallholder agriculture vulnerability and technology-driven agri-food 
transformation optimism to curb hunger through increasing agricultural 
intensification (Mabhaudhi et al. 2019). These have already been shown to 
provide only marginal benefits with deeper impacts on smallholder 
agriculture, increasing dependency on agro-industrial food systems 
(Mabhaudhi et al. 2019). Concomitantly, evaluation of international 
development agency programmes continues to show low levels of 
effectiveness on the sustainability and resilience faced by farming 
communities. Green politics perpetuated the history of ‘white science’, 
pushing technology-based solutions by international institutions and 
corporations with an implicit ideology, power and agenda to reproduce 
agro-capitalist interventions, raising expectations about the opportunity to 
develop African societies without social reforms (Kilby 2019). 


E Advocacy for a decolonial agenda in 
agri-food systems 


More recently, during the commencement of the 2021 UN Food Systems 
Summit, discontented societal and grassroots movements signed a 
declaration, declaring the Summit as a façade because of the dominance of 
corporations’ and foundations’ solutions, in which they unequivocally 
stated (Global Peoples Summit 2021): 
[We] reject the ongoing corporate colonization of food systems and food 
governance under the facade of the United Nations Food Systems Summit [...] 
The struggle for sustainable, just, and healthy food systems cannot be unhooked 
from the realities of the peoples whose rights, knowledge, and livelihoods have 
gone unrecognized and disrespected. (n.p.) 


Activist movements point to the continued co-option of climate resilience 
and food system transformation discourses as central to the narratives 
justifying Green Revolution solutions in the Global South (Clay & Zimmerer 
2020; Mills-Novoa et al. 2022; Patel 2013), with intensification positioned as 
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crucial to ensuring adequate food production in the face of projected 
climatic shocks and population increases (Lal et al. 2015; Müller et al. 2011). 


Activists for climate equity and justice advocate for the importance of 
developing nuanced and contextually appropriate alternatives for 
smallholder farmers. Advocates and farmer movements continue to argue 
for recognition of how climate disasters have exacerbated the vulnerabilities 
of socially and geographically disadvantaged people, including those 
facing discrimination based on gender, age, ethnicity, class, caste, 
indigeneity and disability. The goal is to build adaptive capacity through an 
equity and justice lens. The goal is to build adaptive capacity through an 
equity and justice lens. Poverty and disadvantage compound climate 
change impacts while also increasing local vulnerability to climate change 
impacts. More importantly, because of the socio-historical context, the fact 
that race, culture, ideology and worldview are inextricably linked to these 
discourses needs to be acknowledged. For the first time in more than three 
decades, the International Panel on Climate Change (IPCC) 2022 report 
acknowledged colonialism as a historical and current driver of climate 
change. The report states (IPCC 2022): 

Present development challenges causing high vulnerability are influenced by 

historical and ongoing patterns of inequity such as colonialism, especially for 

many indigenous peoples and local communities [...] Officials and scientists 


from around the globe now recognize the significant role colonialism has played 
in heating our planet and destroying its many gifts. (p. 410) 


E Problem statement 


However, critical movements and scholars continue to argue that current 
conceptualisations of climate equity and justice within food systems 
discourses are inconclusive on the latter fundamentals. Advocacy for 
discourses like food sovereignty that recognise non-dominant frames and 
narratives remain side-lined. The chapter’s main inquiry question is 
influenced by the apparent differential attention given to climate equity 
and justice discourses in the context of transformative food systems. Hence, 
this chapter asks the question, ‘Whose agenda is it?’, as it seeks to uncover 
the prevailing framings of equity and justice within the evaluation of current 
food discourses within the climate adaptation agenda in Africa. 


The chapter’s inquiry focuses on human beings as the targeted 
custodians of food systems, particularly smallholder producers. This 
differentiates the elements of resilience and sustainability from the domain 
of ecological justice, which deals with normative issues pertaining to an 
entire ecological system (Inoue 2023; Wienhues 2020, pp. 16-17). Both 
sentient and non-sentient beings are subject to human actions and policies 
in the environment (Inoue 2023). The chapter assumes that the focused 
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concerns of interest and needs are those of human beings. Importantly, this 
assertion relates only to this chapter’s focus on equity and justice for food 
systems transformation under the climate change agenda. This specific 
view aims to deal directly with these issues without undervaluing the 
ecological elements that fall outside of the scope of this chapter. 


E Inquiry questions 


Upon this historical and contextual analysis, this chapter seeks to investigate 
the agenda of climate justice and equity in the evaluation of climate-smart 
agriculture (CSA) through the interrogation of the following key questions: 


e What are the dominant agri-food systems discourses in Africa? 

e What is the influence of dominant narratives and framings on intervention 
outcomes and impact orientations? 

e How can, if at all, equitable evaluation approaches contribute to 
reframing equity and justice within CSA interventions? 

e What are the opportunities and limitations for evaluators in centring 
equity and justice in the evaluation of CSA? 


E Critical constructivism as a foundation for 
conducting a narrative review 


Critical constructivism paradigm 


This chapter’s investigation is based on the _ critical-constructivist 
paradigm. Critical constructivism combines constructivism and critical 
theory. Using a multi-paradigm model allows for a more insightful 
response to the main inquiry question (Bogna, Raineri & Dell 2020). 
Critical theory is ‘a moral construct that aims to reduce human suffering 
[...] especially suffering caused by conscious choices’ (Steinberg & 
Kincheloe 2010, p. 140). By actively challenging injustices and transcending 
dominant social and political states of affairs that problem-solving 
analyses tend to take for granted, critical theorists argue that the first 
step is acknowledging that problems exist (Steinberg & Kincheloe 2010, 
p. 140). Complementarily, constructivism is based on the understanding 
that knowledge of the world is an interpretation between people, 
institutions or communities (Saurugger 2016). This interpretation takes 
place within a contextualised timeframe. Constructivists assert that it is 
simple-minded and misleading to simply study outcomes of constructed 
processes outside of the processes through which information becomes 
validated as knowledge as well as those that invalidate certain forms of 
information as validated knowledge (Kincheloe 2005, p. 63). Thus, deeper 
inquiry is required to understand why a problem becomes a problem as a 
complicated process in which politics, narratives, concepts and cognitive 
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frames play a crucial role. The investigation includes evaluating the frames 
and narratives that shape food systems discourses as well as the nature 
of the problems, whether the nature of the problem is major or moderate, 
new or recurring, short-term or long-term (Saurugger 2016). 


Narrative review 


A narrative review, as used here, is a process of synthesis that focuses on a 
range of questions - not only those related to a particular intervention’s 
effectiveness - and explores the existing debates and contentions on the 
area under study (Popay et al. 2006). The chapter adopted this synthesis 
approach aligned with the critical-constructivists’ paradigm. Narrative 
reviews tell the story of the findings using both empirical and theoretical 
studies. A search and retrieval process ensured a multi-database search 
strategy, including the African Evaluation Database (AfrED), Wits e-Journal 
Database and Google Scholar, but with a specific focus. The search utilised 
eight search terms - ‘Climate-Smart’, ‘Agriculture’, Justice’, ‘Smallholder’, 
‘Food systems’, ‘Climate change’, ‘Evaluation’ and ‘Africa’ - as core concepts, 
although other concepts were included such as ‘Colonisation’, ‘Race’, 
‘Capitalism’ and ‘Neo-colonialism’, to further refine results in the quest to 
align with the review question. Each concept’s associated synonyms were 
also included in searches in each of the search engines and databases, 
while truncation was used to accommodate any spelling inconsistencies. 


The initial search retrieved 57 documents (including journal articles and 
dissertations) on the agriculture sector, broadly, of which 28 documents 
were specific to CSA in smallholder subsistence farming. The titles, abstracts 
and full texts were then reviewed to identify relevant papers to answer the 
review question. The search was not comprehensive or systematic, as 
narrative reviews do not serve this purpose (Popay et al. 2006). The study 
retrieved key sources that enabled us to tell a story of how the problem is 
defined and how the solutions are framed and operationalised. 


Limitations 


The methodology utilised has three key shortcomings. Firstly, the AfrED 
did not at all times efficiently and effectively produce advanced searches 
extending the search period. Secondly, despite being treated identically, 
the quality and influence of the documents varied during the data analysis 
process. Other studies beyond evaluation studies were included in the 
review to assist in conducting a constructive and critical analysis of the 
topic. These studies could contribute to the volume of studies contained in 
AfrED. Lastly, the low quality of the majority of evaluation reports retrieved 
limits their use in agenda and framing analyses. Nonetheless, these 
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constraints did not detract from the chapter’s purpose, which was to tell a 
different story from that predominantly told in evaluation using a different 
lens and framework rather than being thorough. It seeks to provide a 
different perspective by giving a voice to side-lined perspectives. Thus, 
moving from exploratory to explanatory has insights and supporting 
arguments emerging throughout the data gathering, analysis and 
presentation stages. 


E Agenda-setting and framing analyses in 
evaluative inquiry - what is in a discourse? 


This chapter contends that evaluators pursuing equity in their evaluative 
work should apply critical reflective and constructive reasoning to call 
attention to the way various stakeholders and interventions rely, both 
purposefully and subconsciously, on resources and power to influence the 
agenda to define the boundary of an issue in development. Evaluation is 
defined as ‘any systematic process to judge merit, worth or significance by 
combining evidence and values’ (Better Evaluation 2021). Thus, | contend 
that evaluative thinking in service of equity and justice should be understood 
as a process of sense-giving and sense-making. This requires examining 
evidence to understand the past (Holt & Thorpe, 2007) and context as part 
of sense-making and sense-giving through meaningful interpretation and 
navigation of frames. This is important because the way a problem is 
framed over a historic timeframe may significantly impact the design of 
future interventions relative to the power of those stakeholders who hold 
the power over the agenda. It places the responsibility on evaluators to 
recognise that an idea does not begin with the most obvious source. It has 
a history, and when one begins to trace the origin of a project or problem 
back through time, there is no logical point to stop the process until the 
cause of inequity and injustice is identified (Kingdon 2003, p. 73). Hence, 
this chapter settles first on the history of the problem, politics and policy 
dilemma as exemplified in the introduction. 


Figure 6.2 draws attention to the relationship between agenda and 
framing analyses. Framing is defined as the process by which actors 
construct and represent meaning in order to comprehend stakeholder 
power, relationships, processes and interventions (Goffman 1974; Schon 
1994). Frames are tools that allow stakeholders to ‘find, perceive, identify, 
and classify subjective experiences or information’ (Goffman, 1974, p. 11). 
Framing analysis is a step up towards unpacking context analysis (Scheff 
2005), that allows consensual interpretations of discourse. It is a useful tool 
for institutions and people with varied knowledge, experiences and personal 
histories to analyse common issues and seek to make sense of them from 
their individual or organisational perspectives (Scheff 2005). The frames 
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Source: Author’s own work. 
FIGURE 6.2: Model of inquiry - combining agenda and framing analysis. 


that emerge help actors identify processes and contextualise them within 
a set of values, concepts, discourses or agendas (Goffma 1974). Framing 
analyses can contribute to understanding actors’ underlying logic, 
contestations and values (though not always in a strategic or deliberate 
manner). As a result, the framing process can help to give and make sense 
of complex socio-ecological systems like climate change. 


Frames exist at multiple levels (Lugen 2020). Firstly, at the meta-level, 
the focus is on the existence of frames, the actors who support them, the 
narratives supported and their influence on how equity and justice are 
conceptualised, built over time and implemented within development 
interventions. The success of those actors in pushing their narratives and 
ideologies results in the design of development interventions that contain 
implicit and explicit conceptions of acceptable equity and justice 
considerations. Over time, it is apparent that the current framing of climate 
equity and justice in food systems in Africa acts as a form of manipulation 
of the state of affairs through climate investment in feasible solutions 
placed on the agenda. Ultimately, the acceptability of food systems 
interventions is controlled by the values and ideologies held by international 
development agencies, donors and the political elite. As Kingdon (2003) 
argues, policies and the focus on inequity are rarely placed on the agenda 
to serve the interests of communities and citizens; instead, they are a ‘by- 
product’ of the competition for ‘action’ among powerful actors. This 
requires critical analyses of how different frames and narratives are 
legitimised within the historical and socio-political context taking stock of 
the key drivers of inequity. 
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Secondly, framing can be seen at the conceptual level (Lugen 2020) in 
theories of adaptation and outcomes, as well as in scientific definitions. It is 
easy to see how scientific disciplines play a role in framing because they 
build a shared frame around the research topic and ‘make meaning’ through 
that frame. As a result, frames are revealed in shared key hypotheses, 
methodologies, values or assumptions. At this level, the influence of dominant 
frames manifests in intervention design and the emphasised dimensions and 
criteria, while undervaluing others. This form of control emerges as a defence 
against the exploitation of less powerful actors and a major source of injustice 
and inequities enduring existence (House 2017). In this way, issue frames and 
supported alternatives can perpetuate inequity through promulgating 
programmes and policies that purport to help the disadvantaged, even 
though the supported programmes and policies may adversely affect 
vulnerable groups (House 2017). Deep inquiry at this level has the power to 
contribute to uncovering the dangerous perpetuation of suspect practices 
and programmes. Of particular relevance to the argument and inquiry 
question is what House (2017) cautions against as the ‘white racial frame’ 
and bias pointing to the prejudice and danger it poses to validity, accuracy 
and rigour of the value statement because of its more covert and pernicious 
manner. Deep inquiry at this level has the power to contribute to uncovering 
the dangerous perpetuation of suspect practices and programmes. 


Thirdly, there is the operational level of framing (Lugen 2020), where 
practical adaptation decisions and actions are taken. In turn, these frames 
influence strategies, policies, and processes for adaptation. At this 
level, evaluators need to understand and pay attention to the source of 
ideas and exclusion of actors beyond the ‘usual suspects’, including 
disadvantaged social groups, individuals, social movements and 
indigenous leaders, among others, as sources of novel solutions and 
frames frequently left in the dark. This is important because complex 
socio-ecological systems encompass people, geographies, institutions, 
identities and ideologies intersect to create complex realities that are 
never fully visible to any one individual or from any single point of view. 
Developing stances, judgements and proposed actions based on evidence 
and values of complex issues requires interpreting not only the information 
that is made available but also the information that is available to 
evaluators (Better Evaluation 2021), drawing from various forms of 
knowledge sources and disciplines. 


Importantly, recognising that different actors represent different 
perspectives and ideas but also contribute to expanding understanding of 
the problem, politics and alternatives for solutions. It is a critical aspect of 
evaluative thinking, which is rarely problematised in programme planning 
and evaluation (Archibald 2020). This form of evaluative thinking is 
important for evaluators as it provides a critical and constructive lens to 
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conduct issue framing, examine interests and power dynamics, and 
influence interventions at multiple levels. 


E Characterising smallholder agriculture 
in Africa 


For evaluators to centre equity and justice in evaluation of CSA, it is 
important to understand the peculiar historical and socio-cultural contexts 
in which African smallholder farmers who are dominantly black live in most 
countries. Smallholder farmers range across different age groups, sex and 
abilities across rural, peri-urban and rural contexts. The majority of 
smallholder farming still takes place in peri-urban and rural contexts on 
small farms (less than two hectares constitute most farm holdings), 
characterised by marginal and fertile lands (Beyene 2014; Mukhwana 2003), 
typically farming far from water, off-farm income sources and reliant on 
rain-fed agriculture in many African countries. These farmers also have 
significantly less access to other essential resources required for optimal 
agricultural productivity. The average black smallholder farmer, particularly 
in rural communities, is faced with peculiar challenges of lack of capital 
assets that would enable them to increase agricultural productivity, food 
security and income, access to energy, basic farm tools, water and labour- 
saving technologies. Yet, despite these challenges, smallholders contribute 
significantly to the rural economy and play a significant role in natural 
resource conservation. 


It must be noted that social inequities are differentiated across different 
age groups, sex and abilities. Smallholders in rural, resource-poor contexts 
in Africa can be characterised as a vulnerable group because of the 
combination of vulnerabilities, risks and challenges faced in the face of 
climate impacts in resource-poor communities. The crucial starting point in 
understanding social inequities should be understanding the specific 
farmer, farming systems, relationships and history of the landscape in which 
farmers live and produce. 


Evaluators must examine how these burdens and capabilities manifest 
differently based on individual and group identities to understand the 
differential resource access, systems and the physical environment, that is, 
climate, soil and topography, and the resources and management strategies. 
These characteristics are different for every farmer and farming system in 
different cultures. It is therefore important to understand that these socio- 
economic factors, farm-management and cropping systems and agro- 
ecological systems are embedded in cultural systems. For instance, one 
should not blindly believe dominant narratives of gender inequality and the 
existence of patriarchy in resource access and use because there are also 
African societies governed by matrilineal traditions and matrilineal tenure 
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systems in which females own land and have historically dominated labour 
supply for food production (Berge et al. 2014; Robinson & Gottlieb 2021 
cited in Mwale 2023). Smallholder farmers have differentiated priorities, 
capacities and knowledge which influence their adaptive preferences. At 
the same time, one cannot neglect understanding the differential labour 
and productive capital constraints that may be present in familiar features 
such as age and gender in patriarchal societies, which include care 
dependencies, labour demands for agricultural productivity and household 
keeping. One should recognise that often the challenge is how females are 
able to leverage relationships and connectedness at intra-household and 
inter-household levels to build adaptive capacity and resilience. 


Further, smallholder farmers possess traditional ecological knowledge 
that serves as the foundation of a farming community’s culture, such as 
distinct indigenous kinds of farm-management methods and smallholder 
farmers’ social identities (Ogunyiola, Gardezi & Vij 2022). Again, these are 
not the same because of cultural knowledge, colonial disruptions of these 
practices and forced removals from land to infertile and marginal lands, as 
well as varying levels of spatial climatic impacts. Therefore, the question 
of ‘who is the smallholder’ is not simply a question of socio-economic 
demographics but of the history of the landscape and socio-cultural 
identity that determine farmers’ own priorities, capacities and knowledge, 
which in turn influence adaptation preferences and needs. 


E Reframing climate equity and justice 
in climate-smart agriculture 


The analysis is structured according to the research questions and presented 
in themes identified in the analysis of reviewed literature. There are different 
agricultural approaches promoted in Africa to address climate vulnerability 
and impacts which provide comparable conceptions of climate equity and 
justice. These include agroecology, ecosystems based approach (EBA), 
CSA and food sovereignty. This chapter focuses its inquiry on CSA as the 
dominant approach proposed and evaluated among the retrieved studies. 


Climate-smart agriculture 


Climate-smart agriculture was introduced as a concept in 2010 and promoted 
by numerous development organisations, including the World Bank, FAO, 
USAID and Consortium of International Agricultural Research Centers 
(CGIAR), which combines climate change with intensification (Clay & 
Zimmerer 2020; Lipper & Zilberman 2018). All studies and literature 
reviewed lack a precise definition of CSA, suggesting that a broad range of 
agricultural practices might be considered climate-smart (Williams et al. 2015). 
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Through sustainable agricultural reforms, CSA seeks to link food security 
and climate change. The CSA performance is dependent on the local 
climatic, biophysical, socio-economic and development context 
(Williams et al. 2015). While these considerations are accurate, the socio- 
cultural aspect and history of the landscape are also important, as this 
chapter argues. 


Nonetheless, there is an increasing number of studies recommending 
CSA asan alternative approach for agriculture or large-scale implementation 
in Africa (Chandra, McNamara & Dargusch 2018; Clay & Zimmerer 2020). 
These operations are well-supported by international development 
organisations and national governments. In particular, crop intensification 
programmes are intended to change land use strategies from subsistence 
and livelihood-based systems based on a variety of crops to majority 
cultivation of staple cash crops chosen for their export value (Clay & 
Zimmerer 2020; Magesa et al. 2022). These interventions and 
experimentations are supported by international donors and overseen by a 
system of ‘checks and balances’ to ensure accountability to the central 
governments and non-governmental organisations (NGOs) (Clay & 
Zimmerer 2020; Williams et al. 2015). Even though CSA programmes may 
result in robust transformations that encompass the ‘triple-win’, a single 
focus on one of these dimensions persists, particularly intensification (Clay 
& Zimmerer 2020). This is because discourses play an important role in the 
exercise of power, both in determining who gets what and what gets done. 


Dominant equity and justice frames underpinning 
under climate-smart agriculture 


Evidence mapping of climate adaptation studies revealed that adaptation 
research has been limited in its ability to empirically assess and operationalise 
concepts of equity and justice in practice (Callaghan et.al, 2021). Although 
CSA is still being debated, focusing on injustice as a critical issue provides 
a central point of departure to developing solutions to the equity and 
justice call within a larger historical and socio-political context, including 
identifying, conceptualising and defining requirements for a solution. The 
predominant frames influencing how interventions are designed, 
implemented and evaluated were identified using framing analysis. 
Discourses promoting CSAs focus attention on specific issues. They define 
the extent of a problem, the size at which it must be handled and the 
spectrum of potential solutions in an attempt to control the alternatives 
proposed. As a result, these discourses influence the extent to which 
climate equity and justice goals are pushed. In addition, they influence 
which actors control and have access to resources, both of which lead to 
inequities (Keeley & Scoones 2003). 
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Five justice frames were intuitively identified as being dominant in CSA 
literature: climate equity as (1) ‘distributive justice’ - who will be affected 
by climate change?; (2) ‘procedural justice’ - whose voice is heard in 
decisions?; (3) ‘interactional justice’ - the quality of treatment the 
disadvantaged social groups (elderly, females, youth, people who are abled 
differently) receive when adaptation interventions are applied, and 
outcomes are distributed. Then, (4) ‘intergenerational justice’ - relates to 
the cost of present interventions and policy decisions on future generations. 
Lastly, (5) ‘restorative justice’ - oriented towards healing, often of multiple 
harms, including relational and physical harm to human beings and nature. 
Each frame represents different types of drivers of climate risk and 
adaptation solutions, values underpinning indicators and assumptions and 
intended outcomes. 


O Distributive justice 


Climate-smart agriculture equity considerations tend to focus primarily on 
the distributive justice frame, which limits the sovereignty and autonomy of 
smallholder farmers’ choice of adaptation options at community and 
household levels. Distributive justice focuse primarily on indicators and 
outcomes such as technology transfer and adoption, allocation of climate 
finance, private sector value chain support, incentives, policy frameworks 
and national institutional arrangements. A systematic review of farmers’ 
choices and adoption of adaptation strategies revealed that food production 
is still perceived as the major cause of food and nutrition security (Lipper & 
Zilberman 2018; Magesa et al. 2020). Influenced by this framing, interventions 
still tend to focus on the production-focused approach and enhancing of 
market functions that seek to directly influence food security through the 
increasing intensification of smallholder farming systems (Chandra et al. 
2019; Magesa et al. 2020). 


This is largely because of the basic limitations of CSA and development 
efforts based on the Green Revolution philosophy. For instance, evaluations 
of rain-fed smallholder agricultural systems revealed that one needs to 
prioritise technology and farming systems management adjustments over 
local preferences and needs, leading to CSA efforts diminishing smallholder 
resilience by impairing their sovereignty over land use, reducing livelihood 
flexibility and restricting local resource use (Clay & Zemmerer 2020). In 
part, this is also related to findings from reviews of CSA interventions that 
practices continue to promote global perspectives and policy agendas to 
attract climate finance investments nationally based on simplistic narratives 
of the continent’s high vulnerability, low climate budget and smallholder 
destitution to justify the need for food inputs and assistance interventions 
(Clay & Zimmerer 2020; Williams et al. 2015). While these agro-ecological 
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issues are true, they are not the only issues affecting climate vulnerability, 
nor are they sufficient for assessing risk. 


Several studies acknowledge the fact that assessing resilience in socio- 
ecological change entails paying attention to the undue influence of 
political-economic institutions and agencies, promotion of smallholder 
agency and who is regarded legitimate in defining resilience, as well as how 
climate change is a lived experience that pervades livelihoods (Clay & 
Zimmerer 2020; Taylor 2014). Neglect of these issues promotes focus and 
smallholders’ blind reliance on Green Revolution technology (regional crop, 
hybrid seeds, synthetic fertiliser, land and water engineering) (Clay & 
Zimmerer 2020; Jayne et al. 2010). Rather, there is persistent reliance on 
these interventions, redefined as _ state-led ‘social transformation’ 
programmes, which maintain control by private sector players through 
incentivised market linkages to promote the adoption of high-yielding land 
use methods (Cioffo et al. 2016; Clay & Zimmerer 2020). 


In this way, CIPs and other allegedly novel land use policies in Africa can 
be seen as a continuation of similar ‘suspicious’ initiatives to enhance 
agriculture through technology and policy prescriptions previously 
promoted in Latin America and Asia (Patel 2013) with the intention to 
increase influence and control of foreign private sector institutions on 
smallholder agriculture. More importantly, this results in a continuation of 
colonial thinking. 


O Procedural justice 


This form of justice seeks to address social and political struggles and 
unequal power relations with respect to ‘whose voices are heard?’ at 
various levels, including community and family levels, to investigate the 
susceptibility of smallholder farmers to current and future climate change 
impacts. It aims to address power dynamics. Therefore, evaluation should 
consider and assess local risks, specificities and farmers’ priorities. But it 
also extends to local and national levels to consider which stakeholder 
voices are heard and valued on the scope of types of risks, specificities, 
and priorities that should be placed on the adaptation agenda. In Africa, 
this is limited by the allocation of CSA funding through grant-funded NGOs, 
national government, research centres, academia and regional bodies that 
hold legitimacy over decisions on the adaptation agenda. 


O Interactional justice 


This type of justice is related to whose yardsticks, benchmarks and lived 
experience should be utilised to determine the acceptability of an 
intervention and assessment criteria. Bollen (1989) points out that 
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inconsistencies in defining the use of acceptability concepts can limit the 
development of accurate indicators, assumptions and tools of measurement 
used in evaluating interventions. Subsistence smallholders face multiple 
vulnerabilities, including climate shocks, as well as other stresses such as 
questionable land rights, a lack of access to labour, restricting trade, and 
social reproduction and marginalisation processes linked to race, ethnicity 
and gender (Clay & Zimmerer 2020). Studies revealed that increased crop 
yields across heterogeneous smallholder socio-ecological contexts do not 
necessarily translate to improved food access, as assumed by distributive 
forms of justice (Chandra et al. 2018; Clay & Zimmerer 2020; Magesa et al. 
2022). Thus, the focus on intensification leveraging climate change as a 
rationale for Green Revolution-inspired interventions with little evidence 
that considers the viability of realising climate resilience at intra-household 
levels should be questioned. More evidence is required to identify climate- 
resilient pathways for smallholder farmers that recognise the heterogeneous 
socio-cultural realities and historical contexts. 


Wiliams et al.'s (2008) review reveals that African smallholder farmers, 
governments and policy-makers have differing objectives when it comes to 
prioritising CSA objectives. As a result, CSA practices will ultimately be 
evaluated differently depending on whose values are upheld. Arguably, this 
has linked implications for whether CSA priorities will appeal to investments. 
Further, studies indicate that the private sector primarily comprising large 
corporate seed companies supported by international donors and national 
decision-makers play a big role in investing in, influencing and regulating 
smallholder producers’ land use decisions through the provision of 
agricultural inputs and information in Africa (Patel 2013; Yami & Van Asten 
2017). Thus, what works and how is determined by public sector benchmarks, 
overemphasising value-for-money assessments; so, evaluative assessments 
are conducted to provide information towards this end. 


O Intergenerational justice 


At first glance, prominent movements and narratives on climate equity 
and justice appear to address this within CSA discourses. However, 
evidence shows that it mostly recognises the ecological restoration and 
intergenerational justice aspects aligned with Anthropocene Global North 
narratives, not the restoration of human systems. Questions related to 
spiritual linkages to ancestral lands, belonging and restoration are simply 
not included. In addition, the lack of identification of this aspect of 
intergenerational justice denies that the prevalent issues of worsening 
hunger and food shortages affecting smallholder farmers are an 
intergenerational manifestation of the historical colonialism and neo- 
colonial thinking through land dispossession as well as overemphasis on 
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technology-driven and industrial agricultural agendas in the ‘New Green 
Revolution’ at the expense of much-needed social reforms, indigenous 
and local food knowledge, type of land tenure, access to capital and the 
distribution of land. 


O Restorative justice 


This form of justice seeks to restore relationships to rightness. It seeks to 
repair what is broken in order to restore relationships and power imbalances. 
More often, this relates to expressions of environmental rights enshrined in 
national statutes and Constitutions. But these are the most responsive yet 
restrictive in their form of restorative justice mechanisms. It is also future- 
oriented and unwilling to work within existing systems and structures 
perpetuating injustice to bring about change. Hence, it is often pronounced 
as idealist within global political and institutional systems as well as policy 
structures. Restorative justice aligns itself to solutions that restore the state 
of affairs to its original state before the act of injustice took place. Climate- 
Smart Agriculture discourses do not relate at all to this form of justice, 
particularly where human interests are concerned in Africa. 


These discourses are not suitable in situations where water issues, tenure 
security and land governance are raised, for example, in the case study in 
Zambia where over half of the respondents (56%, n = 145) reported that 
households in their village could not obtain additional land for crop 
production in their village. This is also the case in South Africa, where it is 
becoming clear that the water governance laws and frameworks inherited 
from apartheid make it difficult to discuss the issues of CSA in a changing 
climate without tackling water and water availability. These issues are not 
addressed in CSA discourse or interventions. Instead, it promotes a ‘use 
what you have notion’ without addressing the structural and systemic root 
causes of inequity and justice. As a result, the future livelihood scenarios 
for such farmers may consist of barely surviving or ‘hanging in’, whether 
such farmers can ‘step up’ to adapt better to future climate constraints, or 
whether more of these farmers will ‘step out’ of agriculture (Murray et al. 
2016; Ugwu 2019). 


Upon this analysis, the lack of diversity and inclusion in examining drivers 
of climate change is evident. The role of dominant narratives and ideologies 
on climate change within CSA discourses relative to stated main drivers are 
clearly aligned to global political agendas supported by national and 
regional objectives. The commonly identified drivers of unstable food 
systems and the increasing need for immediate food assistance included 
poverty, economic (high food import prices, poor trade integration), 
environmental degradation and socio-political (conflict and displacement) 
(World Bank 2022) are widely accepted. This is connected to acceptable 
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forms of justice permissible in CSA discourses. The absence of other drivers 
such as race, culture, worldviews and ideology is not widely accepted. 


These drivers are more often a result of colonial dispossession and 
exploitation of food production systems in Africa that have systematically 
and structurally left the poorest more exposed and vulnerable to climate 
variability and economic shocks. In particular, the lack of recognition of 
colonialism perpetuates climate inequity and injustice that broke the 
intimate socio-ecological linkages and interconnections that existed in 
Africa prior to colonialism. The neglect in recognising the historical motives 
of exploiting the environment and subjugating communities over 
generations that drove institutions of colonialism make addressing the 
climate issue and implementing solutions even more difficult, particularly 
in a just and equitable manner. Evidently, smallholder farmers’ interests 
hold the least amount of value and legitimacy in defining desirable 
outcomes and impacts of such interventions. Lastly, the lack of overall 
attention to all these elements of justice in CSA discourses lends them to 
be ‘suspect’ and reproduce many of these uneven benefits on food security, 
poverty reduction and the environment. 


Influence of dominant equity and justice frames 
on the outcome and impact orientation 


The various framings of CSA suggest that it can be understood as a set of 
discourses, or narratives, whose outcomes and objectives are influenced 
by the actors’ fundamental interests, ideas, power and ideologies over the 
political agenda, as well as the activities and objectives sanctioned by 
climate-smart discourses. Now, how do these frames influence and shape 
particular forms of resilience expressed in outcome and impact orientations? 
| argue that outcome and impact orientations are limited by justice frames 
with implications on the paradigms, methods, approaches and questions 
applied in evaluation. So far, the chapter has drawn attention to the agenda, 
influence on dominant narratives, ideologies, power dynamics and 
stakeholder convergence and divergence on framing of CSA discourses, 
how this influences the types of equity and justice frames pursued locally. 


If CSA discourses continue to only pursue distributive justice, outcomes 
and impact will be continually viewed as the increased level of use, adoption, 
scale-up, dissemination and investment in CSA practices. Through this narrow 
focus, the orientation of related CSA programmes reinforces a particular 
body of expert knowledge as valid. Transcending towards procedural justice 
pushes the needle a little to the left, attempting to have orientations 
address or, in some cases, mediate and improve the power dynamics in 
institutional and socio-cultural contexts, including the power dynamics and 
relationships around how decisions are made at multiple levels. 
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Interactional justice pursuits necessitate outcomes and impact to centre 
the perspectives of those who experience injustice and inequity, using their 
lived experiences and history as the benchmarks, baselines and yardsticks 
for measuring success. This means that the evaluation of CSA interventions 
and practices should recognise the multiple vulnerabilities across social, 
human, agricultural, economic and environmental systems in relation to 
their lived experiences and knowledges (Mutenje et al. 2019). Current 
interventions focus on household-level compliance disregarding the need 
for both individual and community sovereignty and how resilience is 
determined by cross-scalar, relational and interactional considerations 
locally (Clay & Zimmerer 2020; Magesa et al. 2022; Mutenje et al. 2019). 
This requires greater attention to the power differentials among individuals, 
households and communities to strengthen their adaptive capacity. 


Restorative and intergenerational justice are aligned with critical 
paradigms, such as critical realism and critical-constructivist philosophies. 
Orientations in these cases would be aligned more to explicit outcomes 
and impacts that focus on undoing colonial and setter-colonial injustices 
and restoring the next generation of African smallholders’ sovereignty. 
These paradigms directly address power and politics, suggesting that it is 
unjust to conduct experiments and evaluative assessments on infertile 
lands where people were forcibly placed to test the productivity of imported 
scientific agricultural innovations reframed as sustainable intensification 
and ecological restoration under the adaptation agenda. 


E Evaluation approaches and methodologies 
contribution to reframing equity and 
justice within climate-smart agriculture 
interventions 


There are growing empirical studies on CSA; majority of these focus on 
scientific and technical issues locally showing marginal benefits of plant 
science, agronomy, soil and water conservation on smallholder farming 
systems (Chandra et al. 2018) utilising scientific measurements and 
modelling tools. Evidence on locally and contextually relevant approaches 
and interventions is still low and reveals moderate improvements in 
resilience in smallholder farming systems. In part, the challenge is that 
these ideas originate from Global North-based agriculture, development 
and climate change institutions using global datasets and models of 
measurement (Williams et al. 2015). The CSA has made many claims as an 
attempt to detach from Green Revolution ideologies, claiming to be focused 
on the ‘whole’ socio-ecological system locally, including restoring 
connections between all individuals and how human food systems affect 
these relationships. There is very limited evidence of this. 
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There is a lack of recognition of the peculiar challenges faced by African 
governments and smallholder farmers in addressing multiple and 
interconnected issues, particularly the nexus between water, energy and 
food in order to achieve the stated CSA goals (Allouche, Middleton & 
Gyawali 2019). Understanding and situating resilience and sustainability 
within the operation of power relations in stakeholder and institutional 
dynamics creates space for value-based dialogues, allowing all actors to 
formulate questions about desirable outcomes and whether and how they 
are privileged over others. Thus, there is a need to begin with integrating 
concerns of epistemologies that inform the design, as well as methodologies 
and approaches used in evaluating CSA interventions by interrogating the 
epistemological issues, the hierarchy of stakeholders’ framing of evaluation 
processes, frameworks, measurement tools used and the questions that 
are asked. 


Socio-ecological evaluation approaches are widely used and accepted 
in conceptualising food systems and adaptation interventions like CSA. It 
is vastly useful and important in evaluating in the context of CSA. Its 
applicability requires a more careful application to confining current 
framings of development by confronting injustices of global climate change 
narratives and ideologies, drawing more attention to how Green Revolution 
thinking still sharply delimits who and what is resilient locally. 


Interrogating distributive justice frames in the 
evaluation of climate-smart agriculture 


Often, evaluators use collected data that examine inequalities in income, 
work roles, reproductive roles, education and farming management 
practices to understand smallholder farmers’ adaptive needs. However, in 
order to uncover the challenges faced by females, disabilities and other 
vulnerable groups evaluators should employ African feminist, intersectional 
and relational systems thinking approaches (Chilisa, Major & Khudu- 
Petersen 2017) which encompass participant-led narrative and visual 
analysis tools (Mapitsa 2023). These approaches and methods are useful in 
identifying farmers’ multiple needs and constraints, as well as to enable 
them to shape their own meaning of how CSA can be developed to meet 
their adaptive needs, capacities and aspirations. In doing so, evaluation 
methodologies must also consider power relations and cultural values 
intrinsic to social change and the political dynamics mediating human- 
environment relations. 


A continuing source of positivism’s power is the dominant conceptual 
frameworks of natural and social science disciplines and the practices they 
legitimise regarding what kinds of adaptive choices should be adopted in 
smallholder farming systems. As climate risks and impacts vary, farmers’ 
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local knowledge should be acknowledged as a valid and helpful instrument 
for adaptation (Chandra et al. 2018; Clay & Zimmerer 2020; Magesa et al. 
2022). Scientific knowledge informed by climate modelling and aggregate 
statistics for all knowledge categories is useful in the evaluation of CSA 
practices (Lemos & Rood 2010). Scientific knowledge is the result of 
scientific effort that develops models, hypotheses, and rules based on 
observations and experimental evidence. This knowledge is heavily used in 
CSA to set baselines, desirable outcome and impacts, indicators, metrics 
and measurement tools which often lacks inclusion of nuanced local 
knowledge and lived experiences. 


Interrogating issues of procedural justice in the 
evaluation of climate-smart agriculture 


Interrogating and integrating procedural justice in evaluation will enable 
evaluators to uncover how distributive justice in intervention designs often 
overlooks and limits how different females are often unable to access and 
use that very distributed justice to enhance their capacity to adapt to 
change. Methodological approaches remain frequently quantitative in 
nature. While important, these approaches cannot fully capture the nuances 
of power, inequality and justice within decision-making processes that 
create and constitute human and social systems and subsequent climate 
change impacts and outcomes. 


There are different versions of reality that come from different privileges; 
even among females these versions must be noted. Often, females are 
presented in evaluation reports as a monolithic group. This will require 
innovation of evaluation approaches that include methods such as 
autobiographic retrospective narratives and participatory coded narratives, 
among other methods, to amplify marginalised voices of local preferences 
and poststructuralist notions that challenge assumptions of universal 
concepts and essential categories of sustainability and resilience to 
challenge current conceptions of equity and justice in programming and 
policy-making. 


As climate adaptation decisions and programmes continue to 
increasingly shape the extent to which smallholders are able to adapt to 
climate change, we must think precisely about the complexity of identity 
and who is involved in adaptive decisions, who benefits from them and 
who is burdened by particular sets of adaptive decisions and the impacts 
of climate change. To ignore these questions creates evaluation activities 
where social actors and organisations remain decoupled from their role 
and responsibilities in the construction of and participation in these 
adaptive decisions. Where the embeddedness of cultural and social 
systems are taken for granted, the cultural blindness of funders, programme 
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designers and implementers remains unscrutinised and unchallenged and 
acts as a path-dependent barrier to the envisioning and building of an 
alternative climate transition in future. 


Interrogating issues of restorative justice in the 
evaluation of climate-smart agriculture 


Thus, greater efforts are required to include key questions that do not lend 
themselves well to approaches such as quantitative modelling and methods 
typically employed in CSA adaptation evaluations to reinterpret the breadth 
of possibilities accessible to people, organisations or society to respond to 
change. This necessitates evaluators to anchor evaluative inquiry on 
appropriate evaluation paradigms that guide the evaluation questions and 
selection of methodologies adopted, knowledge validated and produced. 
Evaluations anchored by indigenous, transformative and Afrocentric 
evaluation paradigms enable the evaluation process to unmask versions of 
reality that support an oppressive status quo and seek to identify structural 
and socio-cultural causes of inequality and discrimination (Chilisa et al. 
2017; Mertens 2007). 


Further, combining the use of a multi-paradigm lens can lead evaluators 
to develop appropriate methodologies and approaches that can uncover 
the framings of CSA discourses and agendas, in addition to proposing 
evaluation questions seeking to unveil how limited justice frames which 
continue to perpetually treat symptoms of the underlying problem. This 
requires evaluators to understand cultures in differing contexts, as contexts 
are not monolithic. As aforementioned, this can be achieved by embedding 
evaluation methodologies in Afrocentric values that promote productive 
community practices, extend and build on indigenous language, spiritual 
and social relationships, self-sufficiency and cultural continuity. 


The importance of decentring the heavy reliance on expert and scientific 
knowledge cannot be overstated, which ignores farmers’ knowledge and 
promotes solutions that are unsustainable both ecologically and socially. 
The question is, how do we include their knowledge in evaluation processes? 
Evidently, as reported in various studies, smallholder farmers in African 
countries do not use scientific information (Chandra et al. 2018; Clay & 
Zimmerer 2020; Magesa et al. 2022). Evaluators must rethink and contribute 
to evaluation methodologies that can better account for the nuances of 
power, equality and justice and their underlying influence on the decision- 
making processes across multiple socio-political scales. One of the ways in 
which evaluators do so is by applying intersectionally-informed methods 
such as a multi-sited critical policy analysis, grounded empirical research 
and ethnographic methodologies that can enable evaluators to understand 
how resilience occurs in settings where political ecology, varied smallholder 
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agency and the biophysical environment converge from the perspectives 
of smallholder farmers and communities (Clay & Zimmerer 2020). The 
social science and indigenous science remain underutilised in reviewed 
studies. 


Employing these methodologies necessitates evaluators and 
commissioners to be open to allowing the evaluation process and 
questions to emerge through the conduct of the evaluation in response to 
local farmer engagement parallel to adaptation processes. Essentially, it 
also requires funders and programme designers to embed evaluative 
thinking in adaptation decision-making and processes. This type of 
thinking will open space to ask critical questions relevant to expanding 
procedural justice that address the socio-ecological dynamics that 
influence who becomes resilient. What is considered crucial and why 
(Clay & Zimmerer 2020), as well as in what conditions beyond scientific 
and technical considerations. 


The inclusion of farmers’ knowledge has the potential to increase their 
adaptive management capacity by using situated knowledge from their 
own farming systems. In order to foster this kind of co-learning and adaptive 
processes, evaluators ought to ask additional questions such as whose 
decision-points are important, and who is the evaluation serving? What 
decision-points does evaluation seek to inform? It is important to recognise 
that individual smallholder farmers’ adaptive decisions and information 
needs are different. Thus, quantitative-statistical analyses cannot be 
substituted for contextual and descriptive information on each farming 
system. Therefore, indigenous data sovereignty should be considered an 
important part of transforming the food system, building resilience locally, 
and gathering contextual information to understand whose resilience is 
being built, in what context, and why. Interventions recognise that the 
increase in climate variability necessitates farmers to have access and 
rights to data. 


Interrogating issues of intergenerational justice 
in the evaluation of climate-smart agriculture 


This is essential as a synthesis of empirical studies shows that farmers’ local 
knowledge is scientifically valid but is primarily used as a tool for 
documentation, despite the fact that farmers’ knowledge has been 
developed through experimentation, adaptation and co-evolution and has 
been culturally transmitted across generations (Ajayi & Mafongoya 2017; 
Juana, Kahaka & Okurut 2013; Ogunyiola et al. 2022). Therefore, an 
epistemological shift in socio-ecological approaches should incorporate 
these critical considerations. Further, where evaluators adopt the paradigms 
and methodologies, it is important to look for opportunities to address and 
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recognise power imbalances. In doing so, look for opportunities for 
reciprocity in identifying root causes, enable farmers to generate and own 
their narratives, challenge its dominant narratives, encourage greater 
expression of socio-cultural identity to support adaptive decisions and 
build local networks for social change processes beyond the evaluation. 


Employing these methodologies necessitates evaluators and 
commissioners to be open to allowing the evaluation process and questions 
to emerge through the conduct of the evaluation in response to local 
farmer engagement in parallel with the design and implementation of 
adaptation processes. This will ensure that evaluation embeds evaluative 
thinking in adaptation processes. This thinking will open space to ask 
critical questions relevant to expanding procedural justice that address the 
socio-ecological dynamics that influence who becomes resilient? What is 
considered as crucial and why (Clay & Zimmerer 2020), as well as in what 
conditions beyond scientific and technical considerations. 


Interventions to achieve resilience and sustainability will include 
venturing into unfamiliar areas, experimenting with new or modified 
strategies and responding to developing patterns in dynamic situations 
instead of ruling out influential and external factors given the range of 
unknown possibilities. This is important for equitable evaluations, as the 
inequities may arise during the evaluation process and may not be known 
prior. Thus, evaluators should build upon evaluation methodologies that fall 
within African feminist, intersectionality and relational systems thinking 
approaches which fall within critical theory. As a result, this can contribute 
to ‘avoids the creation of grand narratives or theories’ (Seigart 2005, p. 155 
cited in Podems 2010) while placing greater emphasis on learning from the 
context and local realities, seeking social, ecological or economic resilience, 
adaptive viability to unknown futures in complex interventions like climate 
adaptation. 


Other approaches, such as theory-based approaches, may reinforce 
inequity and injustice goals by reinforcing the pursuit of outcome and impact 
orientations that are not always in pursuit of justice or limited forms of justice 
pursued. Theory-based approaches have limitations: firstly, understanding 
the problem - understanding ‘problems’ using relational systems thinking 
within the African and community context without boundaries, interrogating 
problems through critical analysis of the sub-systems and key drivers beyond 
dominant narratives and individual interventions to shift how evaluators 
think about the vulnerabilities, risks and capabilities associated with 
adaptation processes, indicators and the causal mechanisms, and unpacking 
the implicit and explicit assumptions. Secondly, limitations in identifying and 
including the root causes of problems and tracing how the problem has been 
experienced or conceptualised in different ways over time. Thirdly, inclusion 
of multiple knowledges dominated in their construction and validation. 
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Evaluators ought to encourage developing project-level change theories 
that encompass ‘on-the-ground’ narratives of change. 


E Opportunities and limitations for reframing 
in pursuit of equity and justice through 
evaluative inquiry 


Evaluators should understand the importance of interrogating politics, 
power and ideology. Evaluation in service of equity should be viewed as a 
political and social change process. By doing so, evaluators ought to 
interrogate the ideologies underpinning past and current food and 
agricultural ‘innovations’ influencing the climate change adaptation agenda. 
This enables evaluators to question the current framings of issues, limiting 
the list of alternatives on the adaptation agenda. Acknowledging the 
multiple agri-food models and approaches implemented within 
communities, outside of the evaluated interventions empowers evaluators 
to understand the politics, power and ideologies nested in intervention 
designs. Funding decisions based on colonial mindsets and outdated global 
perspectives are used to perpetuate dominant yet false food systems 
narratives, which, among other negative impacts adaptation processes 
locally. This will broaden the boundary of contextual analysis in evaluation. 
For foundations and donors committed to equity and justice, exploring 
how to decolonise their operations and food systems investments should 
be top of mind. 


Recognise the importance of inclusion of multiple perspectives and a 
relational systems thinking approach in critical and constructive problem 
analysis. While evaluation practice itself is not a cause of racial bias, it 
manifests in evaluation processes through ‘distortion, co-option and 
corrupting’ of the perspectives influencing problem analysis. Evaluation 
commissioners and evaluators should advocate for those who are 
unsuccessful in influencing the agenda - problem analysis and definition - 
to be consulted in evaluative processes. The climate adaptation space is a 
highly contested arena of stakeholders with divergent views; thus, evaluation 
processes should include these divergent views to open space for more 
critical and constructive perspectives in understanding the problem. 
Provided that food is social and political, pretending that the issues 
presented in the chapter can be addressed in a vacuum is an attempt to 
avoid the harsh realities that are inextricably linked to food: exploitation and 
abuse of smallholder farmers and nations, as well as treatment of non- 
human environments, which has resulted in climate change. 


Evaluators should broaden the scope of indicators and measurement 
tools to capture the nested socio-ecological systems. Evaluators need to 
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recognise the values and ideologies that are inherently embedded in the 
indicators set in assessing the outcomes and impact of adaptation 
interventions. The CSA tends to focus on measuring quantitative indicators 
to mostly account for interventions’ environmental, agricultural and 
economic productivity in different smallholder farming systems. There is an 
increase in the use of artificial intelligence and quantitative modelling in 
assessing the performance of CSA practices. In a few cases, indicators of 
food governance are included, yet the inclusion of social, cultural and 
human elements in the outcome and impact is limited. The inclusion of 
these indicators enables programme implementers and evaluators to 
understand the context and multiple systems. Often, the human and cultural 
dimensions account for the interactional, intergenerational and restorative 
justice frames neglected in CSA discourses. Indicators should account for 
the whole socio-ecological system. It is essential to develop genuine 
participatory and _ learning-oriented approaches for defining the 
conceptualisation of the components of food systems, sub-systems and 
drivers in order to outline the parameters of the local food systems. 


It is important for evaluators to also view evaluative inquiry as part of a 
systematic process of sense-making and sense-giving to account for 
histories of inequity and injustice. Policy analysis and historical stories are 
important learning instruments for evaluation practice, and they should be 
examined and generated as advocates of evidence of all kinds pursue their 
answers. It provides a basis for understanding whose agenda is being 
promoted in interventions in order not to reinforce these biases in the 
evaluation of interventions. However, the chapter recognises that this is a 
steep challenge because of the limited power over evaluations and 
development agendas within covert policy processes. Nonetheless, the 
question, ‘Whose agenda is it?’, is important as the African evaluation 
practice seeks to shift epistemological foundations for evaluative inquiry. 


E Conclusion 


This chapter supports the pursuit of justice, particularly for solutions that 
make it feasible for formerly colonised people to return to their authentic 
selves. However, it recognises the limitation of this in dominant CSA 
discourses. If one were to look back at the communities before colonisation, 
the African population was engaged in agriculture with the capacity to 
provide the necessary food, materials for survival and trade. This chapter 
illustrates the powerful use of agenda and framing analyses in aiding 
evaluators who wish to centre equity and justice in evaluative inquiry. It 
brings attention to the usefulness of these tools to be added to the 
evaluators’ toolbox, necessary to identify multiple influences on problem, 
politics and policy in contemporary African evaluation dilemmas. 
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This chapter calls attention to the critical need to understand and pay 
attention to reinforcing and reproducing problematic ideologies and values 
embedded in the design of interventions. Thus, this chapter also calls for 
action from evaluators to become activists of equity and justice, bringing 
interdisciplinary and transdisciplinary dialogues as well as analyses and 
‘unusual suspects’ into evaluation to draw on the lived experiences of 
smallholder farmers. Arguably, this will increase the depth of empirical 
evaluative knowledge and evaluative thinking in this field towards reframing 
climate equity and justice considerations in CSA. 
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E introduction 


Women in South Africa experience very high levels of different forms 
of violence, including intimate partner violence (IPV), neglect, rape and 
femicide. The South African Demographic and Health Survey found that 


11. The following section is based on a reworking of Amisi et al. (2021b). 
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21% of ever-partnered women have experienced physical violence by a 
partner, and 8% had experienced physical violence in the twelve months 
prior to the survey (National Department of Health, Statistics South Africa 
& South African Medical Research Council 2016, p. 355, 358). Other studies 
estimate that between 43% and 56% of women in South Africa have 
experienced IPV, and 42% of men report perpetrating it (Machisa et al. 
2011). In a widely cited regional study, Gender Links found comparable 
rates of victimisation of women. They reported that a large proportion of 
males (Gauteng 78%; Limpopo 48%; Western Cape 35%; and Kwa-Zulu 
Natal 41%) admitted to committing some form of violence against women/ 
females (VAW) in their lifetime (Gender Links 2011, p. 1). The 2022 police 
statistics recorded 52,694 cases of sexual offences and 41,739 cases of 
rape, an increase of 14% and 14.9%, respectively, on the figures from the 
2021 fiscal year (South African Police Services 2022, p. 44). The number of 
sexual offences could be higher than those recorded by the police, given 
evidence of high levels of underreporting. 


Violence experienced by women has gained attention as local, national, 
regional and global women’s movements organised to place women’s 
issues at the forefront of political, social and economic agendas. The 
growing engagement surrounding women’s varied experiences of violence 
saw debates expand, with concerns expressed about the multiple ways in 
which women and men are subjected to violence. Globally, the concept of 
gender-based violence (GBV) was introduced to recognise that the violence 
that women predominantly experience is a consequence of their gender; 
that is, their being female makes them vulnerable to forms of violence 
which are an expression of men’s power over women. Although the concept 
recognises that males are more likely to be perpetrators of violence 
experienced by females (Coalition of Feminists For Social Change [COFEM] 
2018b), the concept also acknowledged that males can be subjected to 
violence. The term GBV was formally adopted in the United Nations 
Declaration on the Elimination of VAW, an international agreement in which 
VAW is defined as ‘any act of gender-based violence that results in, or is 
likely to result in, physical, sexual or psychological harm or suffering to 
women’ (COFEM 2018b, p. 1). 


The conceptualisation of violence experienced by women as GBV 
recognised that this form of violence has its roots in gendered power 
inequities that exploit distinctions between men and women. It is argued 
that GBV happens and is sustained by patriarchy that tips the balance of 
power in favour of men (COFEM 2018b). However, the adoption of the 
concept of GBV has been contentious, with activists critical that the 
hierarchical, relational power imbalances are overshadowed by an emphasis 
on gender being about men and women (Vetten 2013, p. 149). 
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Gender inequality is considered one of the key drivers of GBV. Harmful 
gender norms that dictate how people of a particular gender are 
expected to behave in a given social context are considered to be at the 
root of violence perpetrated by men against women. Consequently, 
interventions to prevent GBV are often designed to address inequitable 
gender norms and other markers of gender inequality as their strategic 
programmatic thrust (COFEM 2018a). However, the relationship between 
gender inequality and GBV has to be located within an understanding of 
historical structural inequalities which continue to inform experiences of 
marginalisation. For example, in South Africa, apartheid entrenched 
economic deprivation, with poverty remaining skewed along racial lines 
and concentrated within apartheid geographical pockets (Amisi et al. 
2021a). This raises questions about how gender inequality could intersect 
with other social factors to increase GBV in South Africa. Scholars have 
therefore questioned the relevance of interventions focused on 
addressing GBV that are directed at only one set of factors that 
contribute towards the marginalisation and vulnerability of women 
(Amisi & Naicker 2021). 


Despite investment by government and non-governmental organisations 
(NGOs) in interventions to address gender inequality and improve the 
conditions of women in society, GBV remains a considerable problem. This 
chapter will explore the applicability and usefulness of gender inequality as 
a focus of interventions to prevent GBV in South Africa, with its history of 
colonial and racial subjugation. It asks if the current conceptualisation 
of gender inequality adequately explains the persistently high levels of 
violence in South Africa and if perhaps the framing of gender inequality 
should be reconsidered, with interventions incorporating other contextual 
factors that elucidate the relationship between the various drivers of 
violence within a given context. 


This chapter uses ‘What is the problem represented to be?’ (WPR), a 
feminist, post-constructivist policy analysis method to interrogate the 
conceptualisation of gender inequality in GBV interventions and evaluation. 
This chapter will contribute to strengthening the conceptualisation of 
gender inequality and approaches to evaluating the effectiveness 
of interventions to prevent GBV. We apply this approach with the intention 
of questioning, exploring and testing a widely used concept in the GBV 
sector. The intention is not to solve the problem but to question and 
hopefully expand on and clarify our understanding of gender inequality in 
South Africa. The focus of this chapter is on the problematisation of gender 
inequality and GBV within the evaluation of interventions. The chapter 
does not offer a critique, nor does it analyse the evaluation methodology 
of GBV interventions. 
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E Literature review 
What is gender inequality? 


Gender inequality refers to the relational power imbalances that occur 
within political, economic and social institutions in public and private 
spheres of life (Bennett 2009). Women inadvertently bear the most adverse 
impact of these power imbalances, with the experiences of marginalisation 
exacerbated according to intersecting factors such as race, class, geography, 
age, ability and identity (Reddy & Moletsane 2009). 


In order to articulate women’s experiences in defining gender inequality, 
Article 1 of the Convention on the Elimination of Discrimination against 
Women (CEDAW) defined discrimination against women as (United 
Nations Women 2003): 

[A]ny distinction, exclusion or restriction made on the basis of sex which has 

the effect or purpose of impairing or nullifying the recognition, enjoyment or 

exercise by women, irrespective of their marital status, on a basis of equality of 


men and women, of human rights and fundamental freedoms in the political, 
economic, social, cultural, civil or any other field. (p. 5) 


The convention gave recognition to how culture and tradition can restrict 
women’s enjoyment of fundamental human rights. In line with this 
understanding, Unterhalter (2015) defines gender inequality as: 

[...] limits or constraints on the opportunities an individual or a group may have 


to choose and realise the actions, attributes and relationships of well-being they 
have reason to value. (p. 3) 


These constraints can be found in culture and traditions embodied in 
stereotypes, customs and norms which are given legitimate social power 
through a country’s legal, political and economic institutions. The CEDAW 
and other international frameworks propose, among other interventions, 
that countries enact legal instruments that give women power and agency 
in areas such as education, reproductive health, suffrage, marriage and 
work outside the home, among others. 


Such interventions are recommended as the foundation for addressing 
discrimination and achieving equality. Gender equality indicates 
conditions where there is no discrimination or restrictions on the basis of 
sex or gender, where people are free to make choices without the 
limitations set by stereotypes and where their various contributions and 
needs are valued equally. In 2015, in their Progress of the World’s Women 
report, UN Women offered a definition of substantive gender equality, 
which is redressing women’s socio-economic disadvantage, addressing 
stereotyping, stigma and violence and strengthening women’s agency, 
voice and participation. 
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Over time, the concept of gender inequality has broadened to reflect 
the inclusivity of lived experiences of marginalisation. However, with the 
expansion of the concept, there has been a diluted emphasis on relational 
power imbalances. For this reason, the concept of gender inequality, 
although widely used, does not have a single definition. A number of factors 
explain this. For example, Merry (2011, 2016) argues that gender and gender 
inequality are complex and context-specific phenomena (Schmid, Cook & 
Jones 2022). The meaning of the concept tends to shift over time and 
space. Also, as Unterhalter argues, defining gender inequality is not simply 
a matter of abstraction and theory. It is an attempt to describe the lived 
experiences of women in a certain context. She (Unterhalter 2015) further 
argues that this: 

[Rlequires an understanding of the institutional foundations that reproduce 

inequalities and that can support equalities. These institutional foundations 


comprise both political and economic processes, socio-cultural norms, and 
policy and management regimes. (p. 3) 


As observed by Vetten (2013, p. 124), ‘power does not circulate equally 
across institutional sites or within networks’, and within such complexity, 
‘L...] projects to advance gender equality, must be constructed’ (2013). 


In the face of such complexity and conceptual difficulty, how does one 
measure gender inequality? If notions of gender and gender inequality 
are context-specific, who gets to decide if an area experiences gender 
inequality? What measures do they use when measuring levels of inequality? 
What measures do they use once they have intervened to assess if females 
enjoy more equality with males? Conventional measures have tended to 
focus on quantitative indicators for which it is easier to collect data or 
where there are data across countries for comparison. These indices often 
measure indicators such as educational attainment, participation in the 
workplace or representation in decision-making institutions such as 
government or parliament. Qualitative dimensions such as processes to 
create an enabling environment for optimal, quality participation have not 
been distilled into measures to determine ‘gender inequality’. 


Interestingly, with respect to these conventional quantitative measures, 
South Africa measures higher overall than most of sub-Saharan Africa, 
though measuring lower when compared to developed countries. For most 
indicators, South Africa also has a low base for both males and females. For 
example, though South Africa experiences a gender gap of 18.8% in tertiary 
education attainment, this is from a relatively low base. Similarly, in labour 
participation in 2021, 46% of women were participating in the labour market 
compared to 59.9% of men, which means a gender gap of about 13% 
(Gender Inequality Index 2021). However, labour participation in South 
Africa is quite low for both men and women, with an unemployment level 
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of 34% and even higher for young people. Another important measure of 
gender inequality is women’s participation in decision-making institutions. 
That is because women’s participation in key institutions in society is said 
to be more likely to promote policy change towards gender equality 
(European Partnership for Democracy 2019). In South Africa, women enjoy 
46% representation in parliament (Gender Inequality Index 2021). Yet this 
representation has not resulted in the kinds of societal transformations 
that were anticipated. In fact, scholars and activists are acutely aware 
that these technocratic quantitative measures conceal more than they 
are able to reveal (Hassim 2006; Mankell 2012; Vetten & Watson 2009). 
These quantitative measures fail to expose the underlying structural 
barriers that are deeply rooted across all institutions and impact the way 
in which relational power hierarchies are entrenched through individual, 
organisational and broader societal practices. 


Gender inequality and gender-based violence 


Gender-based violence, or violence experienced by women, has been 
called a gendered phenomenon, meaning women experience violence 
because of their gender. This is because GBV is perpetrated mainly by 
men against women as a consequence of the lower status which society 
affords women relative to men. In recent years, the definition of GBV has 
expanded to include all forms of violence experienced because of an 
individual’s gender. For example, Mpani and Nsibande (2015, p. 9) define 
GBV as ‘any harm that is perpetrated against a person’s will that has a 
negative impact on the physical or psychological health, development, and 
identity of the person because of their gender’. Though they acknowledge 
that women are disproportionately affected by GBV, Mpani and Nsibande 
(2015) argue that victims of GBV include men, women and children. This 
expansion in definition has been argued to take away the focus from 
violence experienced by women (COFEM 2018). These contentions 
challenge well-established ideas of how gender inequality between males 
and females drives VAW. 


The relationship between gender inequality and GBV does not seem to 
be a linear one. In some cases, researchers and evaluators refer to gender 
inequality as a driver of violence. In their study in Nepal, Dahal, Joshi and 
Swahnberg (2022) found that women believed that they: 

[...] belong to a culture where women worship their husbands as a god, and 

this might be an important reason for men to feel powerful as a god to exploit 

and abuse women. The discussions put forward the idea that the existence of 
discriminatory beliefs, reinforcement of such beliefs, and a blind following of 

such practices produced differences and violence. (p. 14) 


Interestingly, Dahal et al.’s (2022) arguments also demonstrate the impact 
of context on how gender and gender inequality play out. Religion and lack 
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of access to education for females create conditions particular to the Nepal 
case study. In this understanding, within a patriarchal society, men will have 
more power than women because they have control over resources and 
decision-making rights that are sanctioned by societal norms and practices 
(Javed & Chattu 2020). 


At the same time, gender inequality is also considered a manifestation 
of GBV and that which sustains inequality between men and women. As 
Dahal et al. (2022) maintain, though it is often argued that gender norms 
influence violence, at the same time, violence also shapes gender 
performance with fear, sanction and corrective measures for enacting 
respective prescribed gender functions (Dahal et al. 2022). Thus, gender 
inequality and GBV have a complex interplay where they mutually shape 
each other. Other research shows that gender performance is not limited to 
women, that maleness is also gendered, and males who use violence in 
their relationships with females are also likely to use violence in other 
relationships with males (Langa 2020; Peralta, Tuttle & Steele 2010). In 
South Africa, high rates of GBV occur within a context of high levels of 
criminality, violence in social interactions and interpersonal relationships 
shown in high rates of contact crime and a murder rate of 46 per 100,000 
citizens (South African Police Service 2022). Ratele (2018) and Langa 
(2020) also draw attention to the high vulnerability of young black males 
to homicide and other harmful behaviour. 


What we observe from the literature is that it is not only power 
differentials between men and women that influence men’s use of violence. 
Also, gender inequality, particularly the idea of substantive gender 
inequality, is embedded in the daily experiences of women and not easily 
fixed by changing legislation, representation in institutions or even the 
attainment of education for females. Even in countries considered quite 
equal, such as Sweden, gender inequality in different forms can persist 
(Ville et al. 2023). Could violence be addressed by addressing other 
conditions that contribute to males using violence? 


E Methods” 


This chapter analyses how gender inequality is conceptualised in 
intervention evaluation in South Africa. As a conceptual chapter, it considers 
how gender inequality is framed in evaluations of GBV interventions. Thus, 
this paper is not an analysis of evaluation methods. 


A sample of intervention and programme evaluations included in a 2019 
Evidence Map carried out by CLEAR-AA, the Institute for Security Studies 


12. The following section is based on a reworking Amisi et al. (2021b, pp. 1-51). 
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and the Africa Centre of Evidence was analysed using the WPR analytical 
framework. 


The WPR approach suggests that problems are not self-evident and do 
not objectively exist. Rather, problems are framed in a specific location and 
in a context of existing power relations. This context and framing impacts 
research and policy, as every policy proposal contains within it an implicit 
representation of what the problem is purported to be. By extension, every 
research process contains an implicit representation and conceptualisation 
of a problem as researchers perceive it. Thus, ‘what one proposes to do 
about something reveals what one thinks is problematic (needs to change)’ 
(Bacchi 2012a, p. 21). The WPR framework calls for attention to be paid to 
the importance of power and to the effects of power that are at the centre 
of how problems are defined and acted on. Thus, the WPR analysis not only 
focuses on how problems are spoken of (discourse analysis) but is primarily 
concerned with ‘the outcomes and material implications that arise when 
phenomena are constituted in particular ways as particular kinds of 
“problems” (Bletsas 2012, p. 41). 


Traditionally, the WPR has been applied to policy analysis. However, 
there are a few cases where it has been applied to problem representation 
in research, as we have done in this chapter. For example, Marshall used the 
WPR approach to study how the World Bank shaped discourse around the 
‘oroblem’ of disability and the material consequences of this in how nation- 
states came to understand the problem and acted on it. In this case, the 
unit of analysis was not a national policy but a set of policy proposals that 
the World Bank’s researchers, policy advisors, etc., would have 
proposed (Marshall 2012). Similarly, Vidor et al. (2020) applied the WPR in 
a systematic review of literature to find out how the problem of gender 
parity in physics and physics education was represented and the 
assumptions about gender that underlie research. Amisi et al. (2021b) used 
the approach to analyse how research in South Africa frames the problem 
of VAW and children. Indeed, Archibald argued that evaluators need to pay 
more attention to the question of problematisation and problem 
representation and not only focus on how problems are ameliorated by 
interventions (Archibald 2019). 


Papers reviewed 


The analysis focused on evaluations of interventions that were tested for 
their effect on preventing VAW in South Africa. The initial evidence map 
and evidence review had 73 studies covering VAW and violence against 
children. We excluded all studies focused on violence against children; this 
included studies that targeted adolescent girls. We only included studies 
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where interventions aimed to prevent violence experienced by adult 
females because we were interested in how GBV problematisation affects 
interventions to prevent violence and how these are evaluated. There were 
several papers that reported results on the same programme, that is, 
Integrated Mapping and Geographic Encoding System (IMAGE), Stepping 
Stones, etc. We decided to randomly select one article per programme. 


Sixteen papers were reviewed. Each paper was read, and data were 
extracted using the following WPR interrogative questions, adapted for the 
study (based on the work of Amisi et al. 2021b): 


e How is gender inequality represented to be? 

e What presuppositions or assumptions underpin this representation of 
gender inequality in South Africa? 

e How has this representation of gender inequality come about? 

e What is left unproblematic in the representation of gender inequality? 
Where are the silences? Can the ‘problem’ have been thought about or 
articulated differently? 

e What effects are produced by this representation of the ‘problem’? 

e How or where has this representation of the ‘problem’ been produced, 
disseminated and defended? How has it been (or could it be) questioned, 
disrupted and replaced? 


Data extracted were captured ona tracking sheet and analysed thematically. 
We also analysed the South African National Strategic Plan for GBV and 
Femicide (2020) to answer Bacchi’s six questions. We wanted to explore 
where this problem representation has been produced, disseminated and 
defended? How has it been (or could it be) questioned, disrupted and 
replaced? 


In addition, four structured conversations were held with experts in the 
field of GBV prevention. The conversations added some additional insights 
into how gender inequality is framed, based on where one is positioned 
within the sector - policy-maker, service provider, researcher or community 
member. The individuals were purposefully selected because of the rich 
experience, depth of work and recognition across the sector which they 
receive for the work that they have been doing in GBV prevention. 
Respondents were established senior researchers. The first respondent has 
been designing and testing GBV prevention interventions in South Africa 
for over 20 years. The second respondent was a director of an organisation 
implementing both prevention and response services for GBV survivors. 
One respondent was a senior policy analyst on gender and gender inequality 
who has worked with the South African government. The fourth respondent 
was a senior researcher in violence prevention with a prominent South 
African think tank. 
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E Findings and discussion 


Bacchi’s WPR framework was used to structure the presentation of the 
findings and discussion section. 


How is gender inequality represented to be? 


We observed that gender inequality is often used without an operational 
definition. The meaning is often implied and not spelt out. Most explanations 
of gender inequality describe it as power differentials between men and 
women, particularly when it comes to decision-making within relationships 
or the home, in cases where people are married and in community settings. 
We saw similar difficulties with definitions of gender inequality by the 
interviewees. They indicated that a definition of gender inequality is not 
easily coined and drew a distinction between formal and operational 
definitions. Interestingly, interviewees all emphasised how gender inequality 
is far more readily interpreted with respect to operational aspects. 
Interviewees were more likely to describe how gender inequality manifests 
than what the concept means ideologically. 


Researchers draw on a binary representation of gender that draws on 
sex differences between males and females. They also draw on the social 
construction of femininity and masculinity and the relations between the 
two genders within a patriarchal society. South African society is 
represented as patriarchal and provides the conditions for unequal power 
dynamics between men and women. Similarly, interviewees were also likely 
to talk about the differences between men and women. They recognised 
that gender inequality is evolving because of shifts in how gender is 
understood, the shift from binary gender and issues of identity, and the 
dominance of this perspective in framing and shaping policy responses. 
There is a frustration that is, in some respects, generational, whereby the 
framing of gender inequality is becoming more opaque with a complexity 
that is difficult to translate. 


Gender inequality is represented as a driver of violence experienced by 
women. As one paper argued, hegemonic masculinity and other harmful 
constructs of masculinity that emphasise male dominance and authority 
over women lead men to perpetrate physical and sexual VAW. The dominant 
masculine forms also result in men engaging in high-risk sexual behaviours 
and avoiding health clinics - this puts their female partners at risk (Dworkin 
et al. 2013). Gender inequality is also shown as men’s use of power over 
women. The misuse of power by men over women results in IPV. Men 
use controlling behaviours such as sexual, economic, emotional and 
psychological abuse (Wright, Kiguwa & Potter 2007). 
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Some evaluations recognise that GBV is happening within the context of 
underdevelopment and poor opportunities for both men and women, 
though still arguing that, because of gender inequality, women are at 
increased risk of violence (Pronyk et al. 2006). 


What presuppositions or assumptions underpin 
this representation of gender inequality in 
South Africa? 


There is a sense in which patriarchy as a system of oppression is framed as 
the source of gender-unequal norms and beliefs. Papers argue that men 
benefit from the unequal norms and beliefs as they are attributed greater 
power or higher value. For example, one study of an intervention in a peri- 
urban area argues that there are promising signals that group-based 
training among young men can decrease IPV perpetration. For the 
programme’s effectiveness, men who use violent behaviours should be 
targeted. In describing the prevailing gender norms, they indicated that 
male participants were likely to hold the view that women should stay at 
home and men should earn more and have multiple wives, while female 
participants expressed a desire for freedom to work outside the home and 
for male participation in household duties (Hatcher et al. 2020). Indeed, 
this evaluation highlights the limits of performative ‘equalisation’ or 
representation. The design of the intervention was such that the programme 
assigned female volunteers to work with male volunteers to challenge 
gender norms in the community. However, female volunteers were not well- 
received by male programme participants, which undermined their ability 
to critically reflect on their deeply held gender beliefs. The evaluators 
viewed this as a weakness in the implementation of the programme. 
However, perhaps it shows assumptions about the superiority of feminist 
thought that pushed for female representation even when the context was 
not ready for it, undermining the programme’s very objective. Also, it 
showed a disregard for male voices in programming. Programmes targeting 
women are rarely implemented by men because of the understanding that 
men will not fully understand the female experience, and because where 
women have previous experiences of GBV, being in a programme led by 
men might be retraumatising. Thus, programme evaluators were more 
likely to interpret male responses to female programme facilitators as 
driven by gender-inequitable norms and not explore other factors that 
could have been anticipated and planned for in the programme delivery to 
avoid pushback. 


Another presupposition is that community is the setting where gender 
norms are created and enforced. Thus, community outreach and advocacy 
is considered an effective strategy to shift harmful practices toward gender 
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equity and thereby reduce HIV risk and GBV (Christofides et al. 2020). This 
supposition is informed by research in South Africa and the Global North. 
For example, community mobilisation is a strategy that has been utilised in 
countries like the United States of America (USA) to shift gender norms 
and other societal norms. This focus on community and culture was 
sometimes criticised in conversation with those working within the sector. 
Through these conversations, they recognised that the sector focuses on 
what we think we are able to control, for example, home and community, 
rather than broader systems that perpetuate inequalities, such as economic 
and political systems. 


How has this representation of gender inequality 
come about? 


Most of the evaluations draw on feminist theory. Drawing from this 
theoretical framework, GBV is seen as closely linked to the social, political 
and historical context of women’s subordination in relation to men. For 
example, one evaluation stated that there is a general consensus that 
hegemonic masculinities lead to harmful behaviour, including GBV. The 
way masculinity has been constructed encourages men to exert their power 
on women’s bodies without consequences (Christofides et al. 2018). 


Evaluations also draw on national police statistics to make a case for the 
vulnerability of women to violence and murder at the hands of their 
partners. For example, two studies applied a public health model and used 
the South African femicide statistics to show the health impact of VAW. 
They argue that South Africa’s rates of femicide are relatively high compared 
to other countries, which shows how gender inequality, societal norms and 
hypermasculinities drive women’s vulnerability to death at the hands of 
partners. In 2017, the femicide rate in South Africa was 4.8 times higher 
than the global average rate of 2.6. However, this number must be 
understood within the context of high levels of violence and a high contact 
crime and murder rate. What other factors are driving high levels of murder, 
including the murder of women by their partners in South Africa? 


One study used Ungar’s Social Ecology of Resilience to consider 
interactions between the individual and his or her environment in facilitating 
resilience. While resilience can be considered to be dependent on an 
individual’s personal contributions and, to some extent, personality, this 
framework posits that social-ecological support far outweighs individual 
contributions. Other studies and literature are referenced that emphasise 
the role of religion, family and the community in helping victims of abuse 
to overcome adversity. Parallels are also drawn between the African 
philosophy of ubuntu and Ungar’s Social Ecology; ubuntu stresses 
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communality and the idea that we are all responsible for each other. Such 
frameworks expand the dimensions that are explored as drivers of violence 
and can be helpful in unpacking complex interactions between different 
drivers of violence. How do we understand these factors and their 
interaction with gender inequality or societal norms that shape heterosexual 
intimate partner relationships? 


An important finding from the discussions with sector experts was a 
recognition that debates about gender inequality are not happening in 
South Africa. The interviewees asserted that gender inequality has become 
more peripheral, with little debate and discussion about how gender 
inequality is understood within the current context. Instead, attention is 
fixated on the practical manifestations of gender inequalities. Where 
discussions have been prompted, particularly in the formulation of policy 
responses and interventions, those in the sector recognise that while there 
is a leaning towards being inclusive of the diverse expressions of gender, 
this has often been in isolation of any interrogation of how to feasibly 
translate the policy responses into practice. Also, where practicalities have 
dominated the conversations in which gender inequality is framed, such as 
equal work opportunities or equal pay, there is little interrogation of the 
extent to which the intervention facilitates actual transformation. 


What is left unproblematic in the representation 
of gender inequality? Where are the silences? 
Can the ‘problem’ have been thought about or 
articulated differently? 


In a few studies, there is a recognition that the communities where 
interventions are tested are experiencing high levels of deprivation and 
that both men and women are experiencing marginalisation and a lack of 
resources. Notwithstanding this, there is a simplification of why GBV 
happens in these communities, and the essentialisation of gender inequality 
draws attention to cultures and norms and away from structural drivers of 
violence and crime. But as Dixon (2013) argues, the problem of violence 
and crime in South Africa defies simple explanations. There are high levels 
of violence outside of intimate partner relationships that point to other 
drivers that probably intersect with gender inequality that need to be 
explored. This is supported by Bruce (2022), who argues that: 

Chronic violence and crime show that our society is deeply ‘criminogenic’ - 

meaning that social conditions in South Africa are conducive to high levels of 

offending. Building a safer country depends partly on addressing the socio- 


economic foundations of criminality, which requires giving people access to 
more stable incomes, decent accommodation, education and healthcare. (n.p.) 
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Also, there is a focus on physical forms of violence and not so much focus 
on other forms of violence. This is a perspective that Galtung considered 
inadequate if we want to understand and prevent violence. Galtung argued 
that violence exists where human beings are being influenced so that their 
actual somatic and mental realisations are below their potential realisations 
(Galtung 1969). For Galtung, physical violence was a manifestation or a 
consequence of structural and cultural violence. When societal structures, 
such as the economic system, laws, the public service system, etc., 
interact in such a way that they create marginalisation of certain groups of 
people, this is structural violence. Structural violence is difficult to see 
because it looks as if it has no victims, but when society makes it difficult 
for individuals to realise their rights and, therefore, their potential, that 
society is violent. In South Africa, a country that is supposedly middle- 
income, children go to bed hungry, 27% of children are stunted (May & 
Devereux 2017), and their potential is cut even before they reach 
adolescence. Access to education is skewed towards those who have the 
means to pay and experiences of multiple forms of violence during 
childhood are not only common but likely. The second aspect of Galtung’s 
conceptualisation is that of cultural violence, which exists where dominant 
attitudes or beliefs are used to legitimise direct or structural violence. 
These can be stereotypes and beliefs held by individuals or embedded 
within institutions. This is the case for most people in South Africa. Ggola 
argues that in South Africa, those who escape poverty and marginalisation 
are still subjected to racial and cultural marginalisation where the institutions 
they become part of refuse to see them as fully human, refuse to 
acknowledge the history of apartheid and refuse to acknowledge the 
legitimacy of black pain. This, she argues, is seen in the backlash against 
affirmative action, the demonisation of black economic empowerment or 
the pressure to participate and be absorbed into a culture of rampant 
materialism and consumerism (Gqola 2007). 


Sector experts also stressed the importance and complexity of situating 
gender inequality within a broader context. They challenged the idea of 
gender inequality seen from the perspective of the individual. They argued 
that how gender inequality manifests is determined by the role of a specific 
‘locus’, that is, community, family, work, etc. The power distribution created 
by roles and responsibilities and the value derived from different spaces 
could be vastly different and could even be in conflict with one another. For 
example, in a family where love and care are deemed to be the expectation 
or value-add, how one thinks about gender equality may differ when one 
considers the workspace. So, this tension between individual, community, 
family, etc., may need further exploration. 


Left unproblematic is the link between men’s use of violence and earlier 
experiences of violence. Research by Gould (2015) into the life histories of 
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violent offenders found that most violent men had prior experiences of 
violence that were either unattended to or not addressed, including 
incidences of sexual violence. In their study, Gibbs et al. (2020) found that 
violent men have had extensive exposure to abuse and neglect in childhood, 
have been bullied at school, had a cruel father and were extremely poor. 
Yet, in much of the problem representation, boys’ and men’s experience of 
victimhood is either underemphasised or not mentioned as a potential 
contributor to the likelihood of using violence in adulthood and adult 
relationships. Girls’ and women’s experiences of victimhood in childhood 
and how these experiences manifest in adulthood and adult relationships 
are also rarely interrogated. 


What effects are produced by this representation 
of the ‘problem’? 


The framing of the problem shapes the intervention that is evaluated. It 
could also be that the choice of the intervention to evaluate shaped the 
literature review and how the problem was represented in the papers 
reviewed. 


We observe that where gender inequality was framed as a problem of 
unequal access to resources, the interventions involved the economic 
empowerment of women. This is the case for the paper evaluating the 
IMAGE programme. The underlying theory of change of the intervention is 
that economic empowerment of women would increase their power in 
their intimate partner relationships and that they would have some control 
over resources and, therefore, the power to make decisions. 


Where gender inequality is seen primarily as a factor of norms and 
societal beliefs, the interventions targeted men to address their gender 
norms. Sometimes, they had a community mobilisation component, as in 
the One Men Can programme and the Thohoyandou Victim Empowerment 
Programme in rural Venda. In some cases, the interventions incorporated 
both individual men and community mobilisation components. 


Most of the interventions do not address the cultural and structural 
drivers of violence as defined by Galtung. The issue of race and the history 
of race-based inequalities, and the continued impact these have on black 
communities, is rarely addressed in interventions. Except for IMAGE and 
Steppingstones and Creating Futures that attempt to address economic 
disempowerment, interventions do not address the hopelessness and 
disempowering effect of high levels of poverty and inequality. Even the 
interventions that focus on economic empowerment do not address 
the structural drivers of economic dislocation but focus on individual 
factors that lead to unemployment. Even when interventions succeed, 
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they merely move individuals from unemployment to underemployment, 
where people are still stuck with low wages that are not enough to 
transform their lives. 


This is not a failing of programmes per se, but it highlights the 
stubbornness of the problem of GBV in South Africa and the need to 
question the theories we apply to grapple with the nature of the problem 
the country is facing. This highlights the importance of gender inequality 
conceptualisation considering the empirical and conceptual aspects of 
gender - there is often a tendency to focus on the empirical (debates about 
access to resources). However, the conceptual is also about relations of 
power. ‘Denigration of the feminine naturalises domination and “produces 
even as it obscures” vast inequalities of power, authority and resource 
distribution’ (Peterson 2003, p. 28). 


E Lessons 


The ‘What the Problem is Represented to be’ framework is not a problem 
analysis framework aimed at finding one way to understand the problem or 
to find one solution. It is a problem questioning, ‘an open-ended mode of 
critical engagement’ (Bacchi 2012a, p. 22). The idea is that by critically 
engaging and questioning concepts that are taken for granted, new insights 
can emerge that can inform and possibly improve policies and programmes. 
Based on the analysis, we draw lessons for evaluators and those working in 
the GBV prevention sector. 


Evaluation and evaluators 


How problems are represented influences how we intervene to solve the 
problems. It also influences what we monitor and what we define as success 
or failure when we evaluate the interventions. Yet, as this chapter shows, 
the framing of problems is not apolitical and is about power. Who has the 
power to determine what is a problem and what type of problem it is, and 
therefore, what needs to be done about it? 


Often, problem definition and problematisation is done by programme 
designers, in some cases with some participation of affected communities 
but in other cases in their absence. Community workshops and ‘community 
consultations’ can often be avenues not to truly hear community voices 
and how they see and interpret their lived experiences, but a way to 
‘influence’ their thinking and convince them of the need for a certain type 
of intervention which programme designers and donors have already 
decided. As Archibald (2019) argues, evaluators need to pay attention to 
these issues. Sometimes, programmes are ineffective because they are not 
coherent and relevant to the communities they are being implemented in, 


148 


Chapter 7 


even if they are internally coherent, have been implemented effectively 
(meaning all activities have been implemented as intended) and resources 
have been used efficiently. 


For evaluators, the focus on programmes as the unit of analysis in an 
evaluation can also lead to a narrow definition of the issue. Evaluators are 
often guided by what the programme identified as a problem and what 
they intended to do to address the problem (their theory of change or 
programme design) (Archibald 2019; Podems 2019). This narrows the 
evaluators’ focus. We can fail to see how a problem that manifests in a 
certain community can result from factors outside of the community. For 
example, high rates of GBV ina certain area or community do not necessarily 
mean that the drivers are found in that community. The drivers could be 
poor employment, stress levels in workplaces, exposure to adverse 
childhood experiences, etc., which are produced by decisions made by 
corporates, government and other powerful social systems. By focusing on 
the community, we look for drivers in the community and think solutions 
can only come from within that community, whereas in some cases, we 
need to look outside of the community or the individuals experiencing the 
undesirable situation to find solutions. A public health model with its 
emphasis on understanding drivers at individual, relational, community and 
societal levels is helpful in doing this, though we observed that evaluators 
and researchers often find it difficult to fully apply that lens of analysis in 
their research. It is easier to fall back to factors on one level, probably 
because having a focused intervention locus is easier. The growth of 
systems thinking in evaluation can also help in overcoming this narrowing 
of focus on programmes, though our argument is that we need to question 
how problems are represented no matter which approach we use. 


There has been a lot of talk in the evaluation sector about equity, 
transformation and decolonisation, at both international and regional 
levels, as seen in key publications on the topic from IDEAS, CLEAR-AA, the 
African Journal of Evaluation and the American Journal of Evaluation, 
among others. These various debates all come down to the realisation that 
evaluators’ frames of reference, theories and ways of producing knowledge 
can serve to sustain unequal power dynamics. At its worst, the theories on 
which our interventions are premised can actually prevent real transformative 
change in communities. It can undermine the skills and abilities of those 
experiencing some undesirable situations to take action to address the 
problem. Applying the principles of the WPR approach, questioning 
how problems are represented, which voices are dominant and which 
perspectives are undermined, can go along way in ensuring that evaluations 
and evaluative enquiry identify the systems that are driving social problems 
and therefore better inform the design of interventions that can transform 
such systems. 
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Evaluators also need to pay more attention to the methods they choose to 
use in evaluating programmes. Methods and approaches to evaluating 
programmes are not just technical tools. They reflect how we see the world 
or at least interpret what we observe, what we consider as credible data 
and how that data are made sense of. Indeed, evaluation methods and 
tools shape how societies understand certain types of problems and what 
are considered effective solutions. As Bacchi suggests, ‘It means asking - 
what realities do my methods create and with what effects for which 
creatures and places?’ (Bacchi 2012b, p. 7). Therefore, Evaluators must 
question their assumptions in designing evaluations and the analysis they 
carry out. This kind of self-critique and reflexivity is not just for those who 
are conducting qualitative evaluations but also where quantitative designs 
are used. 


Gender-based violence prevention sector 


Preventing GBV is an urgent need in South Africa. The high level of violence 
experienced by women continually erodes development and negatively 
impacts social outcomes. The analysis in this chapter points out some areas 
that practitioners, evaluators, programme designers and policy-makers in 
the GBV sector need to consider for strengthening prevention work. 


There is a need for more spaces for debates and discussions about 
gender inequality in South Africa. Scholars and practitioners need to 
grapple with the meaning of gender inequality (conceptually) in the South 
African context, taking into consideration the history of the country and 
the high levels of deprivation and disenfranchisement. We found that the 
absence of such debates leads to a fragmented view of gender inequality 
and what is needed to achieve equality. The implications are that gender 
inequality has become ineffective as a conceptual framework to guide 
research, evaluation and the design of GBV prevention interventions. It has 
also been ineffective as a means to rally women’s organisations and drive a 
transformation agenda in the country. Government interventions have 
focused on the attainment of quantifiable changes such as women’s 
representation in the workplace, representation in political parties, etc., 
and there is not enough engagement with the ideological changes that 
need to happen to truly build a gender-equitable society. It is clear from 
our analysis that such a debate cannot happen outside of broader 
consideration of what an equitable country looks like and how South Africa 
recovers from apartheid and builds a society in which all citizens enjoy the 
human rights enshrined in the Constitution. 


Addressing power imbalances between genders and the abuse of power 
in relationships and society is an important aspect of preventing GBV. 
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However, as the analysis has shown, there are many factors, such as race, 
employment status, geography, etc., that intersect with gender inequality 
to reproduce high levels of GBV in South Africa. To address GBV, the 
conceptual understanding of gender inequality and how it contributes to 
GBV needs to be layered. Importantly, this layered understanding of gender 
inequality needs to be translated into how interventions are designed. The 
sector needs to exercise caution against leaning towards operational 
aspects in an attempt to provide conceptual clarity, which, as the chapter 
shows, is unsatisfactory for practitioners who envisage and desire strategic, 
transformative change. The nuances of unpacking violence within private 
and public spheres launch one into the realm of needs and interests, with 
positionality a determinant of whose or which are prioritised. 


Voices of researchers and programme designers are quite dominant in 
the framing of gender inequality, what is considered unequal power 
between men and women and what equality could look like. Voices of 
women affected by GBV are not as dominant in the framing of the problem. 
How do they see and understand inequality, what would equality look like 
for them, and what do they think should be done to address inequality and 
prevent violence? Creating spaces for multiple voices, experiences and 
knowledge forms to inform what is understood by the problem of gender 
inequality and its contribution to GBV can only broaden understanding of 
the concepts and ensure that more interventions are apt in how they are 
designed to transform underlying structures of power. 


E Conclusion 


Historical tensions persist in terms of how to render visible women’s 
experiences of oppression, with conceptual challenges evident in 
programme evaluations as well as among those working within the sector. 
To move forward, it is imperative that we open up debates about how we 
understand gender inequality and its relationship to the forms of violence 
that manifest in public and private spaces. These debates present an 
opportunity to forge new analytical perspectives that reflect the lived 
experiences of those who occupy a given context. Such conversations are 
not going to be easy but are essential to effect change. 
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The role of evaluation 
practice in promoting the 
prioritisation of mental 
health equity on the public 
health agenda in Africa 


Centre for Learning on Evaluation and Results, Anglophone Africa (CLEAR-AA), 
Faculty of Commerce, Law and Management, University of the Witwatersrand, 
Johannesburg, South Africa 


E introduction 


Mental illness is defined as any condition that applies a clinically significant 
disturbance on the thinking, behaviour, perception and emotions of the 
affected person (Addo et al. 2018). Mental illness causes significant distress 
in the affected person; it causes impairment in functioning, negatively 
affects relationships and interactions with other people and is associated 
with the risk of self-harm. More importantly, for a person to be diagnosed 
with a mental illness, they must experience serious functional and role 
impairment for more than two consecutive weeks, with negative 
consequences on their jobs and careers (World Health Organization [WHO] 
2022). In addition to its debilitating effects, mental illness contributes 
significantly to inequality, particularly in socio-economic status and access 
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to basic services (Hailemichael et al. 2019). People living with severe mental 
illness often fall into poverty because of reduced productivity, decreased 
income, loss of employment and increased medical and transport expenses, 
among others (Addo et al. 2018; Hailemichael et al. 2019). Severe mental 
illnesses include, but are not limited to: major depression; schizophrenia; 
anxiety disorders such as generalised anxiety disorder, panic disorder and 
social anxiety disorder; bipolar disorder; post-traumatic stress disorder 
(PTSD); disruptivebehaviouranddissocialdisordersandneurodevelopmental 
disorders such as attention deficit hyperactivity disorder (ADHD); and 
autism spectrum disorder (ASD) (WHO 2022). 


Some of the most common severe mental illnesses affecting African 
people are depression disorders, including major depression, bipolar disorder 
and schizophrenia. Research indicates that depression is the leading cause 
of disability globally and the second leading cause of the global burden of 
disease (GBD) (Habtamu et al. 2019). The GBD is a tool used to give a 
comprehensive picture of the rate of incidence, prevalence, mortality and 
disability from the hundreds of diseases, injuries and risk factors to improve 
health systems and inform policy-making to reduce disparities in access and 
treatment (Lancet 2020). Depression has been associated with decreased 
quality of life (QoL), increased use of health services, higher degree of 
morbidity (disease or symptoms), increased risk of mortality and overall 
compromised health status. In sub-Saharan Africa, studies have found a link 
between the severity of depression and an inability to perform daily tasks 
that are critical to one’s survival (Habtamu et al. 2019). 


Schizophrenia is another debilitating mental illness. It is regarded as the 
most severe of the psychiatric disorders, as it is profoundly disruptive, 
involving the presence of one or more forms of psychosis, including 
auditory hallucinations (voices) and delusions (fixed false beliefs). Mental 
health scholars say that it is one of the most disabling and economically 
taxing disorders. Schizophrenia may have a relatively low prevalence rate 
globally (0.28%) (Charlson et al. 2018); however, its GBD is extremely high. 
It was ranked one of the 12th most disabling disorders among 310 diseases 
and injuries globally in the 2016 GBD study conducted by the WHO (Ayano 
2016). Despite the extent of these issues, African governments spend only 
a fraction of their health budgets on the treatment of mental illness. The 
latter has been linked to the lack of standalone legislation on mental health, 
including no mental health policies in some African countries (Gberie 2017; 
Sankoh, Sevalie & Weston 2018). 


E The treatment gap in Africa 


According to the WHO (2022), one in every eight people in the world has 
a mental illness. However, the proportion of people suffering from mental 
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illness is disoroportionately higher in sub-Saharan Africa: for example, over 
400,000 people in Liberia suffer from mental illness, over 500,000 people 
in Sierra Leone, almost 3 million Ghanaians, one-third of the South African 
population live with a mental illness (2Omillion people), a fourth of the 
population of Kenya (44 million people) and more than 30% of the Nigerian 
population (approximately 6Omillion people) (Gberie 2017). Research 
shows that the GBD caused by mental health disorders is the highest of all 
health problems, accounting for 13% of the total burden of disease 
worldwide (Samartzis & Talias 2019). Yet the availability of mental health 
care facilities and professionals in low- or middle-income countries (LMICs) 
countries, including African countries, is limited. Sankoh, Sevalie and 
Weston (2018) indicate that, in Africa, there are only 1.4 mental health 
workers, such as psychiatric nurses, for every 100,000 people, compared 
to the global average of nine mental health workers per 100,000. 


In addition to the scarcity of mental health care specialists, African 
countries also have a significantly low number of outpatient mental health 
care facilities and even fewer inpatient psychiatric hospitals or psychiatric 
wards in general hospitals with sufficient beds for patients with mental 
illness (Sankoh et al. 2018). In Kenya, for instance, there are approximately 
80 psychiatrists and 30 clinical psychologists, with approximately 250 
psychiatric nurses. According to Gberie (2017), the WHO also mentioned 
that the Kenyan government spends only about 0.05% of its health budget 
on mental health services, and more than 70% of mental health facilities in 
the country are in the capital, Nairobi. This means that there is 
disproportionate access to these services. Similarly, research by the WHO 
found that only 10% of mentally ill Nigerians have access to a psychiatrist 
or mental health care worker. There are approximately only 130 psychiatrists 
in Nigeria, even though depression, schizophrenia and anxiety disorders 
are common. While considerably better, the South African mental health 
sector has slightly more resources to support its mentally ill citizens, 
however, approximately 75% of these people have no access to treatment. 
It has 22 psychiatric hospitals and 36 psychiatric wards in general hospitals; 
however, access is disproportionately linked to socio-economic status. 
Ghana, Sierra Leone and Liberia are among the most poorly resourced 
countries, in terms of mental health care. Ghana is reported to have only 
three psychiatric hospitals and approximately 20 psychiatrists, and 97% of 
mentally ill people do not have access to these services. Sierra Leone has 
only one psychiatric hospital, which can only support a total of 104 patients. 
The hospital has only one trained psychiatrist and three psychiatric nurses. 
Similarly, Liberia also only has one psychiatric hospital, with only 80 beds 
to support the whole country (Gberie 2017). 


It is against this backdrop that the international community determined 
that the most feasible solution to these resource challenges is the integration 
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of mental health care services into primary health care facilities in LMICs, 
including African countries. The studies found on African Evaluation 
Database (AfrED) evaluated interventions aimed at this integration in 
several sub-Saharan African countries. 


E Study aim 


The aim of this study was to explore how mental health task-sharing 
programmes in Africa are evaluated in terms of the study aims, the 
methodologies used, the evaluand, the results and the outcomes derived. 
The purpose of this study was to determine whether evaluation practice is 
designed to facilitate equitable outcomes for patients or people living with 
mental illness and promote the prioritisation of mental health issues on the 
public health agenda in African countries through the provision of relevant 
information. The research question for this study is: 


Are evaluations designed to facilitate equitable access and outcomes to 
mental health care services in Africa? 


E Methods 


Using the AfrED, a narrative review was conducted to gain an understanding 
of the evaluation studies on mental health programmes in Africa. Narrative 
reviews are comprehensive narrative syntheses of previously published 
literature. This type of synthesis reports on the author’s findings in a 
condensed format that summarises the contents of each article or report 
identified as relevant (Green, Johnson & Adams 2006). The AfrED is a 
collaborative project embarked on by the Centre for Learning on Evaluation 
and Results, Anglophone Africa (CLEAR-AA) and Centre for Research on 
Evaluation, Science and Technology (CREST), which aimed to develop a 
knowledge base of evaluation articles, reports and dissertations conducted 
in Africa. The search terms that were used on the AfrED in this review 
include ‘mental health’, ‘mental illness’, ‘depression’, ‘schizophrenia’, 
‘osychosis’ and ‘bipolar’. The focus was on finding literature on evaluations 
of programmes on clinical mental illness. This meant that documents about 
mental illness issues arising as by-products of other diseases or health 
conditions, such as human immunodeficiency virus CHIV) and acquired 
immunodeficiency syndrome (AIDS), substance abuse or pregnancy, were 
excluded from the study. 


In total, the search yielded sixteen journal articles, theses and 
dissertations. This is a finding in itself because if you simply search ‘health’, 
then the AfrED shows that 635 evaluation documents exist in the database. 
This suggests that this small percentage of mental health care evaluations 
shows that there are a limited number of interventions focused on this 
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population group. This demonstrates that this aspect of public health care 
is not prioritised. 


The topic and purpose of this study meant that further screening of the 
literature was done to find evaluation studies on mental health care 
interventions using the task-sharing approach in Africa. An initial screening 
of the abstracts and introductions of the documents showed that the most 
relevant documents pertain to the mental health Gap Action Programme 
(mhGAP), an intervention implemented by the WHO in many African and 
other LMICs, and studies on the Programme for Improving Mental Health 
Care (PRIME), which is aimed at gathering research and evaluation evidence 
on the implementation and scaling up of the mhGARP. Further screening led 
to the identification of a similar programme developed in Africa called the 
Friendship Bench project. The final selection of literature from the AfrED 
included only three relevant studies. 


E Task-sharing interventions in Africa 
Mental health Gap Action Programme 


According to the WHO, a significant proportion of people on the African 
continent are suffering from severe mental illness; however, mental health 
care services are extremely limited, and access is disproportionately 
distributed geographically and socio-economically, as well as being 
structurally hospital-based (Hailemariam et al. 2019). Thus, there is a high 
treatment gap, as more than 75% of people in many LMICs never access or 
receive adequate mental health care (WHO 2008). This is primarily 
attributed to the low prioritisation of mental health care by ministries of 
health in LMICs, who allocate a small portion of their budgets and human 
resources to mental health care. To address this major treatment gap in 
LMICs, the WHO developed the mhGAP in 2008. The mhGAP involved 
implementing and scaling up evidence-informed mental health interventions 
in over 100 countries worldwide. This programme sought to increase access 
to mental health care by integrating it into primary care facilities, general 
health care services and maternal health care facilities (Lund et al. 2015). It 
was determined that this kind of integration would have significant 
advantages, including ensuring that people are provided with more holistic 
health care, increased accessibility of mental health services, including for 
those from rural areas, opportunities for reducing the stigma of mental 
health problems by not being able to differentiate between patients and 
reduced costs (WHO 2008). 


The mhGAP provides policy-makers, health planners and donors with a 
set of clear and coherent activities and programmes for scaling up care for 
mental, neurological and substance use disorders. The mhGAP programme 
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is based on the best available scientific and epidemiological evidence 
about mental, neurological and substance use disorders that have been 
identified as priorities. Priority disorders were identified on the basis that 
they represent a high burden in terms of mortality, morbidity and disability; 
they cause high economic costs or are associated with violations of human 
rights (WHO 2008). The mhGAP was also based on growing evidence of 
the efficacy and cost-effectiveness of specific sets of treatments for priority 
mental disorders in LMICs. However, research found that evidence is still 
lacking on how these specific interventions can be combined into integrated 
packages and delivered in routine primary health care, general health care 
facilities and maternal health care facilities. Furthermore, there is limited 
evidence on the process and impact of scaling up such an integrated MHCP 
for a country, even at a local district level (Lund et al. 2015). The latter 
prompted the development of the PRIME. 


Programme for Improving Mental Health Care 


The PRIME is a project which was established to generate evidence on the 
implementation and scaling up of integrated packages of mental health care 
interventions for priority mental illnesses in primary and maternal health 
care settings, implemented through the mhGAP. The PRIME project was 
implemented across five LMICs, including Ethiopia, India, Nepal, South Africa 
and Uganda. It was established in response to a call for grant proposals from 
the United Kingdom (UK) Department for International Development (DFID) 
in 2010 to establish a research programme consortium on the theme of 
improving mental health services in LMICs (Lund et al. 2015). 


The PRIME is founded upon five guiding principles, two of which make 
this programme relevant to this study on the role of evaluation evidence in 
promoting the prioritisation of equity in mental health care. The principles 
include: (1) a focus on health systems strengthening, (2) working in 
partnerships and (3) focusing on priority mental disorders that impose the 
largest burden of disease, including depression, alcohol abuse and 
schizophrenia. The fourth principle is the ‘use of robust frameworks for the 
design and evaluation of complex interventions’. The Programme for 
Improving Mental Health Care uses the theory of change (ToC) framework, 
drawing on theory-based programme evaluation approaches to develop an 
overarching theory of how mental health care plans can best be designed 
and implemented to influence the intended outcomes. One of the studies 
on the AfrED focuses on the latter component of the PRIME. The last 
principle of PRIME is the reduction of inequities. According to PRIME (Lund 
et al. 2015), the: 


[Blenefits of implementing mental health interventions should be equitably 
distributed, with a particular focus on outcomes in key disadvantaged groups: 
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people living in poverty, women, and people with severe mental disorders. 
The goal should include reducing inequities both in access to services and in 
improved outcomes. (p. 2) 


E Findings and analysis 


Paper one” 
The first article from the AfrED is titled (Adewuya et al. 2019): 


The effectiveness and acceptability of mobile telephone adherence support 
for management of depression in the Mental Health in Primary Care (MeHPriC) 
project, Lagos, Nigeria: A pilot cluster randomised controlled trial. (n.p.) 


The purpose of the study was to evaluate the effectiveness and acceptability 
of adding mobile telephone adherence support to a collaborative stepped 
care (CSC) intervention for primary care management of depression. The 
CSC intervention package for primary health care workers’ management of 
depression includes screening, psychoeducation and problem-solving 
therapy (PST) as the main psychological treatment that was complemented 
by the provision of antidepressants and referral to a mental health care 
specialist when necessary. The study reports on a follow-up intervention 
where the researchers refined the CSC intervention package to include 
mobile telephone support for pro-active adherence management, which 
involved providing information to help the patients manage their health 
and telephonic reminders to adhere to medication and attendance to 
appointments. 


The study was based on the premise that one of the major contributing 
factors to the large treatment gap for depression and other mental 
illnesses in Africa is poor adherence to the prescribed intervention. This 
study used a pilot multi-centred, two-arm, cluster randomised controlled 
trial (RCT), where patients were randomly assigned into either the 
ordinary collaborative stepped care (OCSC) or mobile telephone- 
supported CSC (mCSC) groups. A total of 456 participants were randomly 
assigned to the oCSC group, while 439 participants were assigned to 
the mCSC group. The link to the WHO’s mhGAP intervention is that all the 
health care staff of the ten comprehensive primary health care centres 
(CPHCs) that were randomly selected for the study were trained in 
delivering the CSC intervention using an adaptation of the mMNGAP-IG 
(mhGAP intervention guide) training manual (Adewuya et al. 2019). 


The study was conducted in Lagos, Nigeria, specifically Ikeja, which is 
the largest of the local council development areas (LCDAs). The population 


13. The following section is based on a reworking of Adewuya et al. (2019). 
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in Lagos at the time of the evaluation was 20 million. The city has five 
administrative divisions and 57 LCDAs, each with an average of four primary 
health care centres (PHCs), with one of the four designated as 
‘Comprehensive PHC’. Each PHC caters for approximately 350,000 
residents, with only three medical doctors, ten nurses or midwives, five 
community health officers (CHOs), five community health extension 
workers (CHEWs) and three Pharmacy technicians. The CPHCs also serve 
as referral centres for the other three PHCs in the LCDA. Ikeja is the largest 
and most populous of the five administrative divisions of Lagos (with a 
population of approximately 7 million people) and it has fifteen CPHPCs 
(eight are located in an urban setting and seven in a rural setting). Ten of 
the CPHCs were randomly selected, five in the rural areas and five in the 
urban areas. 


With regards to sample size, the evaluators indicated that even though 
no formal sample size calculation was required because this was a pilot 
RCT, the sample size was calculated by assuming a minimum cluster size of 
70 and intra-cluster correlation (ICC) of 0.05, modest study completion 
rate of 70%. They determined that a total of 385 participants in each arm 
would have more than 80% power to detect an average adherence rate 
difference of 20% (80% adherence rate in the mCSC group versus 60% 
adherence rate in the oCSC group) at a significance level of 0.05. There 
were 15,859 adult attendees who were assessed for eligibility from the ten 
participating CPHCs, but only 965 (6.1%) met the selection criteria, out of 
which 895 (92.7%) consented to participate in the pilot. Recruitment took 
place between October 2014 and April 2015. The mCSC arm had three 
clusters from rural areas and two from urban areas, while the oCSC arm 
had three clusters from urban and two from rural areas. The average 
number of participants per cluster was 89 (range 80-100). The baseline 
characteristics of the sample participants include a mean age of 34.89 (SD 
11.96), 55.4% females, 60.9% were married, 46.0% had secondary education 
or higher and 10.4% had a chronic medical illness. The mean total Patient 
Health Questionnaire (PHQ-9) score of the participants was 16.44 (SD 
4.35), mean total WHODAS (WHO Disability Assessment Schedule) score 
was 14.45 (SD 9.34) and the mean overall QoL score was 2.55 (SD 1.01). 
Participants’ characteristics in the two arms were well-balanced. There 
were no significant differences in the sociodemographic and baseline 
clinical scores between the rural and urban CPHCs. 


The participants were screened for eligibility using the PHQ-9, which is 
based on the DSM-5 criteria for depression with a score range of O-27. The 
inclusion criteria included scoring 10 and above on PHQ-9; those who 
intended to stay in the project for at least eighteen months; were literate 
enough to read either English, pidgin English or any one of the three local 
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languages (Yoruba, Hausa or Igbo) and completed the written informed 
consent form; and were enrolled in the trial. The exclusion criteria included 
children under eighteen-years-old, the elderly (above 60-years-old), patients 
with serious medical conditions or disability requiring specialist care, or 
people having any form of psychosis or being under psychiatric care. 


The findings showed that with the mobile support intervention, there 
was a Statistically significant improvement in adherence to the intervention 
at both the six- and twelve-month follow-ups. The study also found that 
adding mobile telephone support to the CSC intervention led to significantly 
lower scores on the depression questionnaire at both follow-ups. The data 
also showed higher recovery from depression. At the twelve-month follow- 
up, the mCSC group reported a much lower percentage of disability 
compared to the oCSC group. Also, the mCSC group had a statistically 
significant higher percentage of patients with good overall QoL, significantly 
lower rates of referral to the mental health team and loss to follow-up 
(Adewuya et al. 2019). 


While the results of the intervention seem quite positive and promising, 
it is clear from the methodology used that the new mobile component of 
the intervention and the evaluation study did not consider the entire 
population of people living in Ikeja in Lagos with mental illness but instead 
had a very restrictive sampling frame of people who were deemed eligible 
for random selection into the study. This approach is known as ‘sampling 
bias’, and it ‘occurs when some members of a population are systematically 
more likely to be selected in a sample than others’ (Bhandari 2022). This 
approach is problematic because it gives evaluators the discretion to select 
groups of people with a greater chance of success in an intervention. In this 
study, the evaluators chose to exclude people based on their educational 
backgrounds, age and severity of illness. In this way, this study contributed 
to perpetuating inequity for some of the most marginalised people among 
an already marginalised group of individuals diagnosed with mental illness. 
It is evident that the evaluation was designed to show successful outcomes 
over the challenges. 


According to the Department of Planning, Monitoring and Evaluation 
(DPME) (2022), ‘Evaluationcanand should contribute to the transformation 
of current systems through the evidence it generates to guide decision- 
making’. The evidence generated from this RCT is not a true reflection of 
the true potential of the mobile intervention to bring about social change 
to those most afflicted by mental illness in Africa because, through its 
design, this evaluation has only added to the marginalisation and inequity 
experienced by some of the most vulnerable groups in society. The 
limitations of this study show the importance of evidence synthesis of a 
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wide range of evaluations on the same subject, as this kind of evidence 
would be a better fit to inform decision-making. The study excluded 
people diagnosed with severe mental illnesses, which are in fact priority 
mental disorders according to the WHO and, thus, the ones in need of 
urgent treatment. In addition, literature shows that depression is common 
among the elderly in Nigeria, and new-onset depression is higher in the 
elderly in Nigeria than in many other countries (Ojagbemi, Bello & Gureje 
2018), is very debilitating for them, and very little is known about the 
disorder in this population group (Gureje, Kola & Afolabi 2007). Therefore, 
by excluding them, pertinent evidence about this group will not reach key 
decision-makers, including policy-makers. Also, access to care for children 
is limited globally, but even more so on the African continent (Patel et al. 
2013). It is important that evaluations not perpetuate discrimination and 
inequities by placing exclusion criteria that make it impossible to gain 
new information on groups of mentally ill people who are already 
marginalised in many ways. 


This then also draws attention to whether the use of RCTs is the most 
effective approach to evaluating mental health programmes implemented 
in government facilities because the aim of public sector interventions is to 
reach previously disadvantaged and unreachable areas. Public mental 
health is defined as ‘a population-based approach to mental health, to 
improve coverage, outcomes and coordination of interventions provided 
by different sectors’ (Strelitz 2022, p. 3). The inclusion and exclusion criteria 
used in this study (exclusionary bias of evaluation sampling) mean that the 
study did not contribute accurate information on the effectiveness of the 
intervention. According to Bhandari (2022), sampling bias limits the ability 
to generalise the study findings because it compromises the external 
validity of the findings, specifically population validity. This means that the 
findings from biased samples, such as this one, can only be generalised to 
populations that have the same characteristics as the sample. In addition, 
this impact evaluation was not designed in a way that it measured access 
and outcomes that the PRIME interventions were supposed to achieve. It 
was not a well-designed evaluation of what needed to be considered. 
Funders, commissioners and evaluators who contributed to the design of 
this evaluation may be among the many who continue to believe that 
quantitative studies, with their ability to make causal links, are the most 
objective form of knowledge generation, as these evaluators made little 
effort to incorporate qualitative methodologies to understand the 
experiences of the participants, their reasons for adhering to treatment or 
not. The latter is critical information vital for programme designers and 
decision-makers because it may show areas of intervention that need to be 
included in the scaled-up intervention. Equitable evaluation challenges the 
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notion that quantitative-experimental methods should have dominance in 
a field that should be guided by the purpose of the evaluation and the 
needs of the key stakeholders. It has come to be understood that evaluations 
using mixed methods are far more powerful tools than a study using a 
single methodology. 


Paper two" 


The second paper under review from the AfrED is a PhD dissertation titled 
‘Using ToC to design and evaluate complex mental health interventions in 
low- and middle-income countries: the case of PRIME’. The PhD formed 
part of the PRIME Research Programme Consortium. The aim of the study 
was to explore how having a ToC can strengthen the design and evaluation 
of complex health interventions, with specific focus given to PRIME. The 
study involved a systematic review to determine the extent to which ToC 
has been used to design and evaluate public health interventions. In 
addition, a comparison was made between the process of developing the 
ToC in all five PRIME countries, including Ethiopia, India, Nepal, South Africa 
and Uganda. Also, the study reflected on the value of ToC workshops by 
conducting a framework analysis of workshop documentation and 
interviews with facilitators. Further, the researcher explored the development 
of the ToCs within the programme, the implications for the development of 
the intervention, and the choice of evaluation methods. Furthermore, the 
researcher presents a ToC for the integration of mental health care in LMICs 
and provides ten lessons learned from PRIME in terms of the application of 
ToC to complex mental health interventions (Breuer 2018). 


The study showed that the ToC approach has been utilised in the design 
and evaluation of public health interventions since the 1990s; however, it 
has not been used in the process of using ToCs in public health interventions 
and how ToCs have been used. The study determined that ToC workshops 
added great value to the development of ToCs for PRIME because there is 
a collaborative space for stakeholders. This participatory approach ensured 
the development of a logical and structured ToC, a contextualised Mental 
Health Care Plan (MHCP) and stakeholder buy-in. It also found that different 
stakeholders at the ToC workshops contribute different kinds of information, 
as this process involved multiple stakeholders, including mental health 
specialists, researchers, policy-makers, district-level health planners and 
management and service providers. This ensured that the stakeholders 
could provide content on all the areas required for the development of the 
ToC, including the level of existing mental health services; the human and 


14. The following section is based on a reworking of Brueuer (2018). 
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financial resources available; infrastructure; clinical knowledge about 
identification, treatment and recovery of mental illness; the evidence base 
underlying effective treatments and monitoring; and evaluation of the 
PRIME programmes (Breuer 2018). 


Other findings included the importance of taking into consideration 
the hierarchies that exist within the health system in the ToC workshops 
because power dynamics can influence the group; having a comprehensive 
cross-country ToC process can result ina programme theory that is relevant 
for a complex multi-level intervention in different contexts; a cross-country 
ToC can provide a framework with which to map contextually relevant 
interventions and also be used to complement other development 
intervention approaches. The study discovered that developing a cross- 
country ToC provides an approach that ensures that the interventions have 
the same functions; however, they can be developed locally and tailored to 
be contextually relevant. In addition, this study discovered that the ToC 
must be used to complement other methods used for intervention 
development. The Programme for Improving Mental Health Care also used 
additional approaches to develop the interventions, including situational 
analyses; qualitative interviews with district managers, service providers 
and people living with mental illness; costing of the interventions; and 
piloting. Lastly, this study showed that one of the key advantages of using 
a ToC approach for the development of a mental health intervention is that 
the causal pathway is made explicit, and indicators are developed to 
measure each step along the pathway (Breuer 2018). 


The study mentioned that even though the PRIME ToC does not explicitly 
contain references to theories from social science, the ToC does implicitly 
refer to several theories relevant to global mental health, including the 
Tanahashi model of health services coverage, which it draws from 
significantly. The Tanahashi framework reinforces thinking on global mental 
health, which aims to increase the equitable coverage of services. This 
framework identifies five levels of coverage of health services, which are 
adapted to mental health care services in the PRIME ToC and are 
operationalised through the development of specific indicators for 
measurement. The five levels of coverage of mental health services include: 
(1) availability (people for whom services are available); (2) accessibility 
(people who can use the services); (3) acceptability (people who are 
willing to use the services); (4) contact coverage (people who use the 
services); and (5) effectiveness coverage (people who receive effective 
care) (Breuer 2018). 


Figure 8.1 shows the summary PRIME cross-country ToC from Breuer 
(2018). Despite using a theory-based approach, the actual focus of the 
intervention is on the treatment of mental illness, and there is no mention 
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FIGURE 8.1: The Programme for Improving Mental Health Care Cross-Country Summary ToC. 
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of the prevention of or promotion of mental health. Research shows that 
while effective interventions are intended to prevent mental illness from 
developing, to promote mental well-being and resilience and to treat 
mental illness and its associated impacts: ‘there is little provision of 
interventions to prevent mental disorder and its associated impacts, or 
promote mental well-being and resilience’ (Strelitz 2022, p. 3). This is highly 
problematic because studies show the treatment of mental illness typically 
occurs later in life (i.e. adulthood), the first onset of mental illness usually 
occurs in childhood or adolescence, and early intervention has the potential 
to help reduce the severity and persistence of primary disorders and 
prevent secondary disorders (Kessler et al. 2007). 


Evidence from a systematic review shows that many mental illnesses 
diagnosed in adulthood have their origins in childhood and argues that 
(Mulraney et al. 2021): 


Effective prevention and early intervention strategies for mental disorders 
therefore has the potential to significantly reduce the burden of disease globally 
as well as improve lives for individuals with mental disorders. (p. 182) 


The latter literature suggests that it is in fact more equitable for public 
mental health care interventions to also incorporate prevention of mental 
illness and intervention in childhood for at-risk populations because the 
outcome and impact would be far-reaching. According to the WHO (2021), 
only a small fraction of government funding is spent on prevention. If 
PRIME was intended to generate evidence to facilitate the implementation 
and scaling up of the mhGAP integrated packages of mental health care 
interventions in primary and maternal health care facilities, then the limited 
focus of its ToC on the identification and treatment are likely to limit the 
effectiveness of the mhGAP. 


This study and the ToC in Figure 8.1 show that the expected impact of 
the PRIME cross-country ToC is improved health, social and economic 
outcomes for people living with mental illness; however, there is no 
explanation of what these outcomes are. Vague outcome statements in 
a ToC have several negative consequences, including (1) leading to 
unclear and ambiguous thinking about what needs to be done to reach 
them, (2) sabotaging the ability of funders, implementing agents and 
stakeholders to develop consensus about what is important for 
programming and funding allocations and (3) creating difficulty for 
developing a measurement strategy to determine when and if, the 
outcomes have been achieved. People suffering from severe mental 
illness cannot obtain or maintain employment, and some become 
homeless, meaning that change in how these individuals experience 
discrimination in workplaces and in society will not be solved by simply 
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treating their mental illness because this kind of solution places the 
blame on the victim and their condition for their socio-economic 
circumstances and does not hold structural and institutional systems in 
society accountable for their discrimination and bias. This evaluation 
dissertation failed to take note of this and acknowledge it and thus 
reverts to the medical model of intervention. 


One of the biggest issues with this ToC is its overall vagueness; the 
wording is not specific with regards to which ‘medication’ is in the 
functioning supply chain, which ‘services are accessible, affordable and 
acceptable’ and which ‘treatment as intended’ is received. Mental illness is 
an individual condition and requires individualised treatment and services 
for improvement to occur in the health, mental health, social and economic 
situations of people living with mental illness. The focus of the ToC was on 
equitable coverage of public mental health services based on the Tanahashi 
framework, which is commendable; however, equitable access cannot 
solely be focused on the quantity of coverage but also on the quality of 
coverage. One of the issues that creates inequity for people with mental 
illness is the cost or price of treatment, including medication and therapy, 
etc., particularly the best treatment, according to medical experts 
(Christensen et al. 2020). What was required here was a clear indication of 
the treatment packages and service combinations that are theorised to be 
effective for producing positive outcomes for people with mental illness. 
Programme for Improving Mental Health Care hosted various ToC workshops 
with various experts on mental health, and yet the ToC does not reflect any 
of this expertise, knowledge and understanding. 


This evaluation dissertation claims that the design of the intervention 
and the evaluation are based on the theory-based approach and makes the 
statement that ‘Although the PRIME ToC did not explicitly contain references 
to theories from social science, there are several theories relevant to global 
mental health which are implicit in the ToC’ (Breuer 2018, p. 231). The latter 
is highly problematic because it shows that this programme and its 
evaluation approach are based ona ToC but are not based on any change 
theories. Scholars indicate that theories of change must be informed by 
change theories (Reinholza & Andrews 2020). Reinholza and Andrews 
(2020) state that: 


[T]heory of change is project-specific and related to evaluation. It makes 
the underlying rationale of a project explicit, which supports planning, 
implementation, and assessment of the project [...] In contrast, change theories 
represent theoretical and empirically grounded knowledge about how change 
occurs that goes beyond any one project. (p. 1) 


This study shows that epistemological limitations of evaluation practice 
perpetuate inequities for people with mental illness. 
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Paper three” 


While the third article should have been excluded from the study because 
of its primary focus on mental disorders in people living with HIV, the 
task-sharing approach of the intervention is the reason why this article is 
relevant to the current chapter. The third paper under review from the 
AfrED is a journal article titled ‘Using a theory driven approach to develop 
and evaluate a complex mental health intervention: the Friendship Bench 
project in Zimbabwe’. This article is said to have been inspired by the 
PRIME, reported on by the PhD dissertation summarised earlier. It 
discusses a similar study that sought to generate evidence on how to 
deliver complex interventions to reduce the treatment gap for mental 
disorders in LMICs, particularly in sub-Saharan Africa. The study was 
focused on a complex mental health intervention called the Friendship 
Bench project. This programme consists of a cognitive behaviour therapy- 
based (CBT) intervention that also uses PST for the treatment of common 
mental disorders. The intervention was implemented by trained lay health 
workers (LHWs) employed by the city health authorities in Harare, 
Zimbabwe. This article described the way in which the ToC approach was 
used to design and evaluate a successful cluster RCT and a scale-up plan 
(Chibanda et al. 2016). 


This article indicates that a theory-based approach was used to design 
the Friendship Bench programme, which was implemented for eight years 
prior to being evaluated and that this approach was also used to guide 
the evaluation of the programme. In addition, a cluster RCT was used to 
evaluate the programme. The evaluation was conducted in two workshops 
focused on the cluster RCT illustrated as 3a-g in Figure 8.1. The study 
stated that eight ToC workshops were held with various stakeholders over 
a period of six months. Like the PRIME studies, the focus was on four key 
components of the programme including: (1) the formative work, (2) 
piloting, (3) evaluation and (4) scale-up. The key stakeholders who 
participated in the ToC workshops included the City of Harare Health staff 
and the research team from the Friendship Bench project. A ToC map was 
developed as part of the process during the workshop, with defined 
causal pathways leading to the desired impact. The primary reasoning 
provided for the development of the ToC causal pathway to impact was 
to ensure that the intervention leads to a reduction in the common mental 
disorders symptoms among those receiving care through the Friendship 
Bench RCT (Chibanda et al. 2016). 


15. The following section is based on a reworking of Adewuya et al. (2019). 
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Figure 8.2 shows the final ToC that was developed after eight workshops 
and ten small group meetings. Two themes were of specific interest in the 
ToC workshops: (1) political buy-in and (2) capacity-building, particularly 
among LHWs, including the development of an acceptable, user-friendly 
and feasible psychological intervention. The workshop discussions also led 
to the modification of the existing PST with an emphasis on PLWH. One key 
highlight from the evaluation was that obtaining political buy-in Cla) 
through the participation of the Minister of Health was important as the 
minister reinforced the notion that ‘mental health care packages should be 
integrated into existing primary health care services’ (Chibanda et al. 2016, 
p. 5). The latter is related to the assumption of the ToC that all 60 clinics 
would appreciate the need for mental health integration for it to lead to an 
expansion of the initiative (2b). 


The evaluation, however, showed that in practice, the latter is difficult to 
implement. A quote from a participating LHW stated that (Chibanda et al. 
2016): 

We see a number of mental health cases, but we don’t have the capacity and the 


adequate time to provide structured counseling for these patients because we 
have to take care of everybody else at the clinic. (p. 6) 


The latter finding shows the value of evaluation in providing evidence of 
the feasibility of the task-sharing approach. This is linked to the fact that 
the most important input identified in the ToC is funding. Without adequate 
funding to hire additional LHWs, the effectiveness of this intervention is 
limited. The latter finding is so important because it shows that international 
agencies such as the WHO, which is seen as an authority on health care 
globally, may have the best intentions in terms of finding solutions to 
problems in LMICs, including African countries, but are not very successful. 
The problem with this is that much of what is seen as solutions for 
developing countries was conceptualised in contexts far removed from the 
political and socio-economic circumstances that have major impacts on 
interventions and the beneficiaries. 


Another important finding from the evaluation was that even though 
sufficient evidence from previous evaluation studies on the Friendship 
Bench project supported the feasibility and acceptability of the intervention; 
the views, buy-in and input of patients were a requirement to move to the 
next stage in the causal pathway (Chibanda et al. 2016). The latter is critical 
for promoting equity in developing societies and an important consideration 
for equity-focused evaluations (Bamberger & Segone 2011): 

If equity-focused evaluation is to be truly relevant to interventions whose 

objective is to improve the well-being of worst-off groups, the equity-focused 

evaluation processes must be used to foster wider participation of worst-off 


groups, facilitate dialogue between policy-makers and representatives of worst- 
off groups, build consensus, and create ‘buy-in’ to recommendations. (p. 12) 
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E Discussion 


This chapter shows that there are two extremes in the way in which mental 
health care interventions that focus on task-sharing have been evaluated in 
African contexts. On the one extreme are poorly designed (and thus, 
implemented) evaluations, and on the other extreme are contextually and 
culturally responsive evaluations that include all key stakeholders in all 
phases of the evaluation, including the design. This chapter showed an 
example of the one extreme, which is the problematic use of single-method 
studies involving RCTs or other quantitative-statistical methods for the 
evaluation of development interventions (Hariton & Locascio 2018), and on 
the other end of the spectrum is the unproblematised use of the ToC or 
theory-based approach, which is deemed a good model for evaluating 
complex interventions because of its consultative/participatory nature, and 
these are designed to help clearly articulate underlying assumptions from 
the beginning of an intervention, and because they are designed to answer 
evaluation questions about (INTRAC 2017; cf. Reinholz & Andrews 2020): 
[W]hat worked (by measuring or assessing the changes brought about by a 


development intervention), but also why and how it worked (by examining the 
processes that led to those changes). (p. 1) 


The review conducted in this study showed that there are clear limitations 
to the use of both approaches, particularly in the context of public sector 
interventions aimed at supporting disadvantaged and marginalised groups 
of people living in African countries. 


The proponents of RCTs regard it as a useful approach to evaluating the 
effectiveness or impact of an intervention or treatment because it is 
believed to produce high-quality levels of evidence through the reduction 
of bias stemming from the randomisation process. It is viewed as a 
scientifically rigorous tool for examining cause-effect relationships between 
an intervention and an outcome and can thus produce an internally valid 
impact estimate (BetterEvaluation 2021; Hariton & Locascio 2018). Some 
scholars even go as far as to assert that this is the easiest and most effective 
approach to use for conducting an impact evaluation (Glewwe & Todd 
2022). While its design may have some merits, many of its limitations make 
it unsuitable for the assessment of government interventions aimed at 
distributing resources to disadvantaged people from under-resourced 
communities or for use in equity-focused evaluations. 


Firstly, the selection of evaluation designs should be determined by the 
evaluation questions, not by the belief that experimental designs produce 
better results. Over the years, studies have shown that some questions 
cannot be accurately answered using an RCT design (Cook & Thigpen 
2019). The RCTs are not appropriate in LMICs such as African countries, 
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where context is a critical factor in the outcomes produced by an 
intervention and important for understanding the reasons the intervention 
was successful or failed to produce the desired outcomes. The RCTs are 
quantitative in nature and thus overlook the context that provides an 
insight into the environment in which those with mental illnesses live. 
Secondly, there is the issue of ‘mixed treatment effect’, which means that 
simply because effective outcomes are found in one group and not the 
other group in an RCT does not necessarily mean that the intervention will 
have better outcomes for all individuals in that group or other groups in the 
future (Cook & Thigpen 2019). Evidence from a multitude of studies has 
shown that several individuals in both the intervention and control groups 
do improve, but that several individuals in both groups do not. 


The most relevant and well-known limitation of RCT designs, however, is 
their inability to produce external validity. This refers to the fact that the 
results and conclusions of an RCT cannot be true for other groups of 
people, in other settings, at other periods of time. This is because of the 
unavoidable differences between the study conditions and sample and the 
contexts and populations to which the study aims to generalise the findings 
to. There is a common misconception that the findings from the RCT can 
be generalised to all patients, treatment/intervention environments and 
cultures (Cook & Thigpen 2019). This chapter showed that in its effort to 
control for confounding variables and increasing statistical effect or power, 
the study reviewed in this chapter showed that having a homogenous 
sample of diagnostically uniform mental health care patients is not 
representative of the actual demographics and complexity of the population 
of people living with mental illness in that area or country. According to 
Cook and Thigpen (2019), the less simple patients who are often excluded 
from participating in RCTs are called ‘marginal patients’, and many of the 
requirements needed to improve the internal validity and control for 
confounding bias in an RCT result in creating an artificial setting that does 
not closely match the real-world environment (Cook & Thigpen 2019). 


It is clear from this chapter that equity is about inclusion, not only the 
inclusion of citizens’ voices through participation in key evaluation 
processes but also about having inclusive evaluation approaches and 
designs that do not contain exclusion criteria. Evaluations should be 
inclusive in terms of the kinds of people who are sampled, which is why, 
instead of random sampling, equity-focused evaluations should use census 
sampling. Census sampling involves gathering data from every member of 
a population (Sriram 2011 ). One of the key advantages of this sampling 
strategy is that unlike other sampling strategies, which only select a subset 
of the total population for inclusion in the study, a census reduces the risk 
of sampling error. A sampling error is a statistical error that can occur when 
an evaluator selects a sample that does not represent the entire population 
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of data (Lavrakas 2008). Sampling error means that the evidence produced 
from evaluations using RCTs can lead to erroneous decision-making by 
governments, showing that while evidence use has become a key discourse 
and endeavour in the field of evaluation in Africa, the scrutiny of the quality 
of that evidence should be an even bigger discussion because using the 
wrong evaluation approach or design has the potential to produce findings 
that could contribute to furthering inequities. Hailemariam et al. (2019, p. 2) 
aptly stated that ‘inadequate care for people with SMD [severe mental 
disorders] in LMICs contributes to disability, poverty, marginalisation, 
premature mortality and human rights abuses’. 


On the other extreme is an over-reliance on the theory-based approach 
to evaluation, which is evident in this chapter as two of the three studies 
reviewed used this approach for the design of the task-sharing interventions 
and their evaluation. The argument is that the ToC approach is useful for 
evaluations because it can adequately capture the complexity of developing 
countries and their varying complex contexts (Breuer 2018). The latter may 
be true if the approach is appropriately applied. This chapter showed that 
both the studies reviewed did not involve the actual use of change theories; 
one of the studies explicitly stated that ‘Although the PRIME ToC did not 
explicitly contain references to theories from social science, there are 
several theories relevant to global mental health which are implicit in the 
ToC’ (Breuer 2018, p. 231). These studies show that what theories of change 
in fact do is make the underlying rationale of a programme explicit, and this 
is meant to support the planning, implementation and evaluation of that 
programme. However, scholars argue that ‘Grounding projects in change 
theory allows change agents to draw on existing knowledge and to better 
contribute to our collective knowledge about how to achieve meaningful 
change’ (Reinholz & Andrews 2020, p. 1). 


The reviewed studies showed that all theories of change are a range of 
hypotheses about how change is expected to occur, and these hypotheses 
are then investigated and revised along the programme lifecycle. Some 
scholars, however, argue that interventions aimed at transformation should 
ensure that theories of change are grounded in change theories and in so 
doing ensure that the programme is contextualised and more likely to 
succeed. According to Reinholz and Andrews (2020), change theories are 
theories that represent generalised knowledge about how changes work, 
some of which are mature theories because they have a strong empirical 
basis. This is an important finding because it suggests that if equity-focused 
evaluations intend to use the ToC approach to effectively capture the 
complexities of the contexts in which the interventions are implemented, 
change theories stemming from the respective disciplines should be used. 
Evaluators need to understand change as occurring within a complex 
system because the context of a programme will affect the outcome and 
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impact of the intervention. Tested change theories are relevant for 
understanding the context and, therefore, should assist evaluators with 
identifying and characterising aspects of the system and culture and 
describe how these might influence how change in an intervention occurs 
(Reinholz & Andrews 2020). 


The problem with using a simple linear unidirectional ToC or logic model 
approach is that it does not acknowledge or address any systemic issues 
because systems are complicated, with many components and actors 
which can be identified in advance and whose contribution to the ToC 
should be articulated (Rogers 2017). Green, Sim and Breiner (2013) state 
that ‘Complex systems are a configuration of interacting, interdependent 
parts, connected through a web of relationships, that form a whole greater 
than the sum of its parts’ (p. 255). Based on this definition, it is possible to 
regard mental illness as a complex system in itself because medical and 
psychological specialists indicate that the exact cause of a mental illness is 
unknown; however, it is a result of a complicated combination of genetic, 
psychological and environmental factors (Sachdev 2023), all of which are 
systems in themselves. For instance, according to Paris (2022), ‘a genetic 
system is essentially a set of instructions that dictate our genetic makeup 
- what we look like and how we interact with our environment’. The child 
development theory by renowned American psychologist, Urie 
Bronfenbrenner, which is called the ‘Ecological Systems Theory’, views the 
development of a child ‘as a complex system of relationships affected by 
multiple levels of the surrounding environment, from immediate settings of 
family and school to broad cultural values, laws, and customs’ (Mcleod 
2023, n.p.). 


Systems thinking shows that the use of simple unilinear theories of 
change is inadequate for understanding the complexities within which 
mental illness emerges and persists, and thus, acomplex range of treatments 
is required to support people living with mental illness. The multi-layered 
complexity of mental illness requires evaluation to take a systems approach. 
According to Green et al. (2013, p. 257), ‘Evaluations that do not consider 
systems dynamics or conditions will likely miss aspects of the intervention 
and its environment that influence the intervention’s operation and success’ 
(p. 257). Evaluation practice has increasingly become aware of the 
importance of systems thinking, but this is primarily with regards to 
government systems and organisational systems; however, in psychology 
and psychiatry, there are important theories that refer to the diverse 
biological, neurological, psychological and socio-cultural environmental 
systems that influence people’s mental well-being or lack thereof. The 
focus of theories of change cannot then solely be on the medical services 
required to treat people living with mental illness, but equity-focused 
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evaluation should aim to address each interacting system to truly bring 
about change for affected people, and this begins with incorporating 
empirically tested theories of change. 


E Conclusion 


Evaluations that aim to contribute to equitable outcomes for people living 
with mental illness need to show an understanding of the illness from the 
medical perspective but also take into consideration the societal structures, 
institutions, cultures and norms that exacerbate the mental conditions and 
socio-economic circumstances of affected people. At the beginning of this 
chapter, literature was cited that described how people living with mental 
illness are highly likely to lose a job and have a decreased income. So, 
instead of reinforcing the value of interventions that solely focus on the 
illness and that assume that by addressing the illness alone, the socio- 
economic circumstances that these people find themselves in will 
automatically change, equity-focused evaluations need to consider how 
institutional and structural change can be facilitated. This can be done by 
providing evidence that supports increasing the accessibility and integration 
of people living with mental illness into mainstream society, including 
workplaces and schools, among others. Also, equity-focused evaluations 
need to consider how institutional and structural change can be facilitated 
by providing evidence that supports increasing the acceptance of people 
living with mental illness by bringing awareness to what it is and what it is 
not and the importance of support facilities or services in mainstream 
institutions but also changes in the way in which working and learning is 
conducted. For equity to occur, evaluation needs to contribute to building 
a society that is adapted to integrate and include people living with mental 
illness into every sphere and one that allows them to thrive, not simply one 
that integrates mental health services into mainstream public health 
systems. 


Some of the approaches that consider issues of equity, inclusivity, power, 
socio-historical context are approaches that fall within the ‘transformative 
evaluation paradigm’ (Fish 2022): 


This paradigm is deeply rooted in the human rights agenda. Therefore, the ethical 
implications include the conscious inclusion of people generally excluded from 
mainstream society, making it applicable to culturally complex communities. 
This paradigm also pays attention to power issues in the research or evaluation 
process, including when designing and planning, implementing and using the 
research or evaluation findings. (p. 8) 


Examples of these approaches and methods include participatory 
evaluation, emancipatory evaluation, human rights-based evaluation, 
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empowerment evaluation, transformative evaluation and an indigenous 
approach originating from Africa: the participatory rural appraisal (PRA). 
The PRA was recently renamed Participatory Learning for Action (PLA); it 
is an evaluation approach that enables smallholder farmers to analyse their 
own circumstances and to develop a common perspective regarding 
natural resource management and agriculture with others in their villages 
(BetterEvaluation 2021). 
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E introduction’! 


This chapter draws on the author’s PhD thesis, critically exploring how 
the Global Fund as a global health partnership for monitoring and 
evaluation (M&E) has (un)intended impacts on the tracking of key M&E 
performance indicators for public health programmes in Zimbabwe. The 


16. Sections in this chapter represent a reworking of Grand, Z 2022, ‘Exploring the effects of Collaborative 
Global Health Partnerships in the Ministry of Health and Child Care’s Monitoring and Evaluation systems 
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chapter explicitly explores how the partnership discourse emphasising 
mutuality and equity-based collaborative governance systems leads to 
missed opportunities to identify and correct evaluative inequalities in 
Zimbabwe. Drawing insights from the new public governance (NPG) 
theory (Osborne 2006, p. 381) and critical discourse analysis (CDA) 
(Fairclough 1989; Van Dijk 2005), the chapter provides insights into the 
uncomfortable and under-discussed negative effects of collaborative 
partnerships involving resourced global health partnerships (GHPs) from 
wealth countries and resource-constrained local partners in developing 
countries. 


This chapter provides an overview of partnership literature, evaluation 
and inequity as the basis and benchmarks for problematising partnership 
practice through support from the Global Fund to fight against HIV, 
tuberculosis and malaria (GFATM) in Zimbabwe. This chapter draws insights 
from the social constructivist paradigm and CDA, arguing that partnership 
discourses promote the vaunted benefits of formal autonomy, freedom, 
knowledge-sharing and informed deliberation, which have become the 
greatest threats to promoting equity in evaluations as they promote a false 
sense of mutuality and equity between the GFATM and its associates on 
the one hand and the Ministry of Health and its local health partners on the 
other. 


Documentary reviews and key informant interviews with M&E staff 
purposively selected for their knowledge of the subject provided the views 
that form the basis of the discussions in this chapter. The conclusion in this 
chapter is that partnership discourse hides and depoliticises the effects of 
power and, therefore, prolongs the possibility of equitable evaluations as it 
gives the false impression of a self-fulfilling discursive logic of shared 
decision-making, participation and informed deliberation among the 
partners in Zimbabwe. As a result, (un)declared contests are inherent in 
this partnership arrangement, thereby failing to fulfil the key tenets of 
mutuality, trust, and organisational identity, which form the basis of true 
partnerships (Ansell & Gash 2008; Brinkerhoff 2002; Emerson, Nabatchi & 
Balogh 2012; Vangen, Hayes & Cornforth 2015). 


E Background and context of the 
Global Fund in Zimbabwe 


Zimbabwe's health M&E policies have evolved significantly because of the 
involvement of global health partners like the GFATM. This transformation 
has shifted from the alma Ata-based primary health care (PHC) evaluation 
policies of the 1980s, aimed at rectifying colonial-era imbalances, to the 
more liberal selected primary health care (SPHC) approaches of the 1990s. 
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By the turn of the millennium, GHPs had become the preferred method of 
public health governance (Buse & Harmer 2007, p. 259). 


The GFATM, established in 2001, plays a prominent role in this 
landscape, promoting transparency, accountability and inclusivity in health 
programmes. It mobilises technical expertise and financial support from 
the private sector and various governments. In Zimbabwe, GFATM began 
supporting M&E systems in 2001 through a round-based system, later 
transitioning to the new funding model in 2014. This support expanded 
tracking of prevention and treatment services across the country, 
particularly for HIV, with GFATM funding many recipients of antiretroviral 
therapy (ART). However, budget reallocation in 2012 resulted in a funding 
gap for antiretroviral (ARV) procurement in 2014 (Ministry of Health and 
Child Care 2013). 


The GFATM’s 5-year budget from 2016 allocated a significant portion 
toward strengthening the routine M&E reporting system in Zimbabwe (Jain 
& Zorzi 2017, p. $97). Initially managed by three principal recipients (PRs), 
Zimbabwe’s GFATM programme shifted to United Nations Development 
Programme (UNDP) management in 2009 because of economic challenges 
and governance concerns highlighted by an audit of round 5. The additional 
safeguard policy (ASP) was implemented in May 2009, excluding the 
government as a direct recipient of GFATM funds. 


The UNDP became the PR for all GFATM grants in Zimbabwe, extending 
to the new funding model from 2014. Sub-recipients, including government 
entities, civil society organisations and UN agencies, played critical roles in 
grant execution. 


The GFATM’s 2017-2022 data use and improvement framework aimed to 
build sustainable national M&E systems, emphasising country ownership, 
integrated health management information systems (HMIS), data analysis 
and use and stakeholder engagement beyond the Ministry of Health. The 
country’s coordinating mechanism played a vital governance role in 
Zimbabwe, emphasising partnerships, effective representation, transparency 
and good governance practices. It ensured balanced representation, 
engaged key populations and oversaw PR performance. 


Despite the support, Zimbabwe's national health M&E and information 
system faced challenges. Given the substantial reliance on the latter, the 
government’s dual role as a donor and implementer raised questions about 
the nature and impact of its partnership with GFATM. These questions 
warrant scrutiny to understand the partnership’s unintended effects on the 
health M&E system. 


In summary, Zimbabwe's health M&E policies have evolved with GFATM’s 
involvement, but challenges persist, raising important questions about the 
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nature and impact of the partnership between the government and a major 
global health partner. 


E Conceptual framework 


The following section discusses GHPs within the NPG and CDA frameworks. 
The section also discusses evaluation and inequalities as it applies to the 
arguments raised in this chapter. 


Conceptualising global health partnerships 


Global Health Partnerships in this chapter are understood within a 
network governance framework in which state and non-state actors 
emphasise the desire to achieve shared goals in specific areas of global 
health (Buse & Walt 2000, p. 550). Similarly, Carlson (2004, p. 5) proposes 
an operational definition of GHPs as a collaborative relationship among 
multiple organisations sharing risks and benefits to achieve shared goals. 
Unlike Buse and Walt, Carlson’s conceptualisation focuses on the goals 
and formal structures instead of the actors. The Department for 
International Develooment (DFID) and the World Bank share a similar 
conceptualisation with Carlson, viewing partnerships as including ‘a 
mechanism to assess success and make adjustments’ or ‘an agreement to 
work together to fulfil an obligation or undertake a specific task by 
committing resources and sharing the risks as well as the benefits’ (Buse 
& Walt 2000, p. 550). Likewise, Buse and Harmer (2007, p. 259) focus on 
‘institutionalised initiatives to address global health problems, in which 
public and for-profit private sector organisations have a voice in collective 
decision-making’. 


The World Health Organization (WHO) describes a partnership in the 
health sector as a means to ‘bring together a set of actors for the common 
goal of improving the health of populations based on mutually agreed roles 
and principles’ (Kickbusch & Quick 1998, p. 69). The definition considers 
agreement on key principles as a crucial factor without destabilising the 
balance of power between the parties to enable each to retain its core 
values and identities. According to the WHO, the ultimate goal of ethical 
partnership principles is to achieve public health care gains and beneficence, 
to avoid maleficence such as ill-health and to ensure partner autonomy and 
equity (Buse & Walt 2000, p. 550). 


The preceding conceptual discussions locate GHPs within the NPG 
theory (Osborne 2006, p. 381). Osborne conceptualises a new governance 
framework that emphasises inter-organisational collaborations based on 
trust or relational contracts, with institutional and implementation 
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structures that promote plural planning and decision-making. The chapter 
elaborates the theory within the framework of four governance models, 
namely the collaborative governance model (Ansell & Gash 2008), the 
integrated framework for collaborative governance (Emerson et al. 2012), 
collaborative governance (Vangen et al. 2015) and the government-non-profit 
partnership model (Brinkerhoff 2002). The NPG theory assumes consensus- 
oriented stakeholder collaborations emphasising trust, mutual interests 
and respect for partner autonomy (Ansell & Gash 2008; Brinkerhoff 2002; 
Emerson et al. 2012; Vangen et al. 2015). These fundamental assumptions 
provide the building blocks and scaffolds for a better understanding GHPs’ 
contribution to evaluative inequalities as unintended effects. However, the 
key assumptions of the NPG theory tend to be its major weaknesses in 
collaborations that involve partners with unresolved mistrust backdated to 
the colonial period. As a result, the chapter further draws from the CDA as 
a complementary framework that analyses the framing and structure of 
global health care development terminologies to reveal its high politics and 
effects on the partnership in practice. 


In a broader context, CDA represents frames, narratives and normative 
appeals in interactive communications and the underlying ideologies, 
public philosophies and values they represent (Barlow & Thow 2021, p. 2). 
The chapter hypothesises that partnerships for M&E use specific discourses 
to hide their underlying ideologies, philosophies and values, thereby 
perpetuating inequalities. As aresult, this complementary framework brings 
the needed dynamism to decode the hidden language that forms the basis 
of prolonging this inequality. According to Van Dijk (2005, p. 352), discourse 
analysis aims to ‘understand, expose, and ultimately resist social inequality’. 
It provides a critical view that identifies power relations and struggles of 
power and the deployment of discursive activities to construct and maintain 
unequal power relations (Yazdannik et al. 2017, p. 4). Furthermore, CDA’s 
analytic amalgamation of discourse as the actual text (description), as 
discursive practices (interpretation) and as social context (explanation) 
(Fairclough 1989, pp. 24-26) shows its superiority in reaching parts that 
other theories and methods cannot reach (Shaw 2010, pp. 200-206) 
thereby providing a comprehensive understanding that exposes and informs 
strategies to resist social inequalities in partnerships ultimately. Hence, this 
chapter draws on these strengths to reveal and expose the use of language 
to conceal the simmering contestations that perpetuate the colonially 
inherited inequalities in evaluative practice in Zimbabwe. 


Conceptualising evaluation and inequities 


Drawing from the Organization for Economic Co-operation and 
Development (OECD) evaluation, this chapter describes evaluation as a 


181 


Mechanisms and strategies perpetuating evaluative inequalities in policy practice in Zimbabwe 


systematic and objective assessment of ongoing or completed projects, 
programmes or policies (Zall Kusek & Rist 2004, p. 12). The focus is on 
design, implementation and results. This process aims to determine the 
relevance and achievement of objectives and to promote efficiency, 
effectiveness, impact and sustainability. The process aims to provide 
credible, heloful information that feeds into evidence-based decisions. The 
process also involves documenting good and bad practices and lessons 
learned in the repetitive learning process to benefit local beneficiaries and 
funding partners. Evaluations draw lessons from beyond direct project 
support. The need for accountability in donor funds utilisation mainly 
drives the increase in demand for evaluation practices in Africa (Mapitsa, 
Tirivanhu & Pophiwa 2019). Whereas development partners provided the 
technical and financial support for evaluative capacity-building in countries 
like South Africa, Uganda, Kenya, Benin and Ghana, there remains a paucity 
of government-led and funded equity-based evaluation initiatives in Africa 
(Mapitsa et al. 2019, p. 62). 


The term ‘inequity’ refers to the disparities between population groups 
that are avoidable and unfairly generated and maintained by what Evans 
and Peters (2001) have termed ‘unjust social arrangements’ that offend 
common notions of fairness (Whitehead et al. 2006, p. 3) and keeps a 
particular group of society in deprivation. While perspectives on what 
constitutes inequality may exhibit some regional variation, there exists 
significant shared consensus on the matter. Scholars have raised concern 
over inequality and inequity driven by the unprecedented encroachment of 
the private sector into large, previously government-reserved parts of 
sectors such as social services and infrastructure but are now dominated 
by the public-private partnership (Picciotto 2015). The level of inequity and 
inequality remains dire in Africa because of a combination of weak 
governance systems, with the world’s poorest 48 countries having 
combined GDPs equivalent to the wealth of the three wealthiest males in 
the world (Coates 2004, p. 250). 


E Literature review 


The following section briefly presents relevant literature demonstrating 
mechanisms and paradigms through which health evaluative inequalities 
are perpetuated. The focused review provides the basis for acomprehensive 
understanding of the arguments. 


The partnership paradigms in policy and practice 


This section identifies and discusses four literary perspectives in the 
partnership discourse in global development. These are pragmatic- 
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instrumental, critical-ideological, critical-governmentality and critical- 
constructivists. 


Pragmatic-instrumental literature 


Scholars in this category view the partnership as a valuable and necessary 
arrangement that is unproblematically implementable. The literature 
assumes that partnerships empower local communities to implement 
initiatives based on mutuality, trust and reciprocity (Brinkerhoff 2002, 
pp. 22-23). The scholarship suggests managerial interventions and 
capacity-building, adherence to policies, regulations and guidelines to 
address partnership imbalances. This perspective’s shortcomings are the 
failure to anticipate and acknowledge the effects of inherited colonial 
imbalances and undeclared interests in the partnership. 


The critical-ideological literature 


Scholars in this category include Abrahamsen (2004), Van Dijk (2005, 
p. 352) and Fairclough (1989, pp. 24-26), who argue that partnerships form 
part of unspectacular strategies to advance neoliberal ideology through 
discourse and ideas. The strategies do not have to rely on aggressive 
physical or coercive programmes but on persuasive and attractive 
processes, including education, religion, entertainment, training and 
development of managerial procedures like M&E. Likewise, Baaz (2005) 
argued that partnership languages are rhetorical disguises or (mis) 
representations that rebrand the old-style paternalistic intentions of 
colonial projects. The partnership language carefully depoliticises and 
nullifies opposition to dominant economic interests through language. 


The critical-governmentality literature 


The scholars subscribing to the critical-governmentality view partnerships 
through the Foucauldian perspective of knowledge and power. Michel 
Foucault argued that power and knowledge work in complementary and 
relational ways. As a result, no one has a monopoly over power as it does 
not reside in specific individuals and institutions. The critical perspective 
identifies the unspectacular deployment of subtle, complex and productive 
bi-directional workings of power through discourse and ideas as evidence 
of influence from afar through cooperative rather than coercive measures. 
The strategies appear to empower yet restrain local action in poorer 
countries through technically depoliticising the governance of 
development (Barnes 2011, p. 38). The scholars draw inspiration from 
works by Li (2007) and Abrahamson (2004) to identify how partnerships 
work through soft power strategies to (re)educate and (re)configure local 
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decision-makers’ habits, aspirations and desires in ways that benefit and 
promote neoliberal narratives. The aim is to achieve conditioning and (re) 
producing modern and self-disciplined citizens who subscribe to the 
neoliberal common sense by enlisting them as responsible agents of their 
liberal development. 


The critical-constructivist partnership literature 


The critical-constructivist perspective views partnerships as contextual 
encounters in which partners construct their understanding of obligations 
in the partnership. The ontological core assumptions of this perspective 
acknowledge the complexities of meaning creation in social and political 
phenomena and that because of the inherently social nature of (f)actors, it 
is impossible to understand partnerships in aid policy and practice unless 
we know their ideas and interests (Hay 2002, p. 254). Likewise, knowledge 
generation and understanding of partnerships shape the actors’ meanings, 
beliefs, preferences and actions within their broader context. However, this 
perspective’s limitation is its erroneous assumption that we can interpret 
and understand the partnerships outside its communicative processes. Yet, 
knowledge creation is sometimes dialogical rather than predetermined. 
The scholars argue that knowledge as a social phenomenon ‘does not exist 
out there’ - it is co-created. 


Soft power mechanisms in global health 
partnership governance systems 


The following section discusses policy agenda setting, norm diffusion, 
rhetoric and discourse control, framing, conceptual boundaries and 
performativity of M&E artefacts as soft power mechanisms in partnerships. 


Policy agenda-setting and institutional control 


The policy agenda-setting process is a contested space. Scholarly work on 
agenda setting has shown that the process is an art (Rothman 2011, p. 53) 
and science, suggesting the need for tact and craft competency. The 
scholar demonstrated how global health partners rely on expert, scientific, 
and technocratic power to influence local development agendas. They 
capitalise on their international reputation, legitimate and expert power 
and discourse control as attractive tools to influence the local M&E policy 
agenda. Taylor and Harper (2014, pp. 214-216) demonstrate how the GFATM 
successfully convinced the local partners to ensure HIV, tuberculosis (TB) 
and malaria are on the national agenda in Uganda, convincing the partners 
to move from the unsupported sector-wide approaches. 
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Similarly, Armstrong et al. (2019) warn that the external influence of Global 
Fund structures on local agenda-setting in Malawi, Tanzania and Zimbabwe 
risked drifting into a ‘political theatre’ because of a lack of genuine 
collaborative engagement with the local partners in budgeting and other 
procedural processes. Likewise, Cummings et al. (2018, p. 735) illustrate 
how GHPs emphasise ‘pluralist-participatory’ discourse at the policy and 
agenda-setting stage and a ‘techno-scientific-economic discourse’ at the 
policy implementation level. Khan et al. (2018, p. 219) discuss how GHPs 
in Cambodia and Pakistan use evidence-based rhetoric to finance 
predetermined HIV agenda items ahead of other pressing challenges like 
mental health. These examples illustrate how GHPs divert local priorities to 
focus on their agenda. 


Normpreneurship and diffusion in global health 
partnerships 


The concept of norm diffusion involves setting common-sensical and 
undisputed, generally accepted standards through circulating ideas at the 
global level. Through ‘normalisation’, ‘influential knowledge producers help 
transform a rough concept into a widely accepted transnational norm 
based on expert knowledge, detailed definitions, and statistical exercises’ 
(Nay 2014, p. 210). The GHPs, like the World Bank, are renowned for 
initiating and circulating influential global ideas. As a result, it acquired the 
status and reputation as a ‘knowledge’ bank and ‘norm’ entrepreneur 
(Knack et al. 2020; Nay 2014). The World Bank and the OECD played a 
major role in ‘normalising the concept of “fragile state,” referring to 
countries experiencing severe instability from weak governance, conflict, 
and severe poverty’ (Nay 2014, p. 211). The concept was transformed 
through successive definition exercises and expert knowledge. Other 
scholars like Sastry and Dutta (2013, p. 23) and Smith (2018, pp. 6-7) invoke 
anthropological and sociological characterisation of ‘doxa’ to denote a 
society’s taken-for-granted, unquestioned truths to define a higher level of 
normalisation influence of GHPs on potentially contested concepts like 
partnerships. The implication of doxa on local agenda setting is that GHP- 
imported ideas become unquestionable and difficult to contest even if they 
perpetuate evaluative inequalities at local health systems. 


Rhetoric and discourse control 


Rhetoric and discourse control are closely related to norm diffusion and 
agenda setting, as GHPs can apply the strategies simultaneously. Rothman 
(2011, p. 49) demonstrates how soft power, like discourse control, is a 
practical means and mechanism to influence international relations. 


185 


Mechanisms and strategies perpetuating evaluative inequalities in policy practice in Zimbabwe 


The technique involves the foregrounding of key messages and 
backgrounding unfavourable conversations. This process unleashes rhetoric 
against some actors to align with the desired discourse. Khan et al. (2018, 
p. 218) observe that GHPs, like the World Bank, and WHO applied inter- 
sectoral leverage strategies in Pakistan and Cambodia to determine how 
the recipient country was perceived globally, influencing the countries to 
align with desired health policies. Negative coverage of these countries 
had the undesired effect of affecting their tourism sectors. 


Framing of monitoring and evaluation issues in 
policy and practice in LMICs 


It is a powerful linguistic tool that relies on texts, symbols and meanings to 
influence policy and practices in LMICs. Rushton and Williams (2012, p. 148) 
demonstrate how framing linguistic, cognitive and symbolic power to 
identify, label, describe, interpret and address problems plays a determining 
role in policy and practice. Similarly, Fukuda-Parr (2016, pp. 49-51) discusses 
the MDGs and SDGs, asserting that framing determines the definition of 
issues, the explanation of causes and the justification of policy responses 
and priorities. Fukuda-Parr makes an important observation and 
demonstrates that the framing of most MDG targets and indicators had the 
unintended effect of marginalising ongoing strategic processes for people 
empowerment and transforming economies with adverse impacts on 
poverty reduction and females’ political voices. Similarly, Shiffman and 
Shawar (2022) discuss how framing global health issues could shape 
differences in levels of priority of attention and resources they receive from 
global health organisations. Through a review of scholarship on global 
health policy-making processes to examine the role of framing in shaping 
global health priorities, Shiffman and Shawar identify the influence of three 
framing processes - securitisation, moralisation and technification. With 
‘securitisation’, they refer to an issue’s framing as an existential threat, 
‘moralisation’ as an ethical imperative and ‘technification’ as a wise 
investment in science (Shiffman & Shawar 2022). 


Monitoring and evaluation of conceptual 
boundaries 


Conceptual boundaries involve a specific ‘position taking’ which occurs 
when actors overtly or covertly relate entities separated by a boundary 
(Kislov, Hyde & McDonald 2017, p. 1426). In this context, boundary work 
refers to the conceptual demarcation of knowledge creation, decoding and 
transmission in various contexts or communities. The boundaries could 
include how scientific knowledge is distinguished from lay or non-scientific 
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pursuits. The GHPs, as ‘boundary organisations’ frame, diffuse and set 
global M&E knowledge production agendas because of their privileged 
epistemic and legitimate power as international organisations. Peters 
(2016, p. 14) and Mueller-Hirth (2012, pp. 664-666) demonstrate how 
expatriates and international NGOs influenced the type of local information 
reported to external audiences in a global project in Angola and South 
Africa. In Cambodia, Aveling and Jovchelovitch (2014, p. 38) observed 
that ‘conceptual representation of self and others’ created boundaries 
between the GFATM and its local partners ina collaborative HIV programme 
leading to the privileged preoccupation with quantitative targets and 
indicators as the local partners chased the performance-based targets set 
by the Global Fund. The mechanisms of influence included expatriate staff 
using rhetoric and discourse control, framing in the translation of global 
scientific information into formats useful for local policy-making, and 
mediating conflicts (Gray 2016; Nay 2014; Rushton & Williams 2012). 


Performativity and monitoring and 
evaluation artefacts 


In the previous discussion, despite its limited value in capturing local 
experiences, the ubiquity of the logical framework provided insights into 
some of the GHP M&E artefacts that perform rather than capture and report 
the local dynamics of M&E. The list of M&E artefacts playing influential 
performative roles in M&E includes GHP guidelines, protocols, plans and 
frameworks. Using an example from Tanzania, Coultas (2020, p. 96) 
provides a dialogical case illustrating how international development 
agencies encourage local practices that perform the logic of rigid 
predictability aligned to international reporting rather than meeting local 
needs. Regardless of the difficulties in collecting locally relevant data, their 
donor contractual requirements provided limited options to adapt the 
donor M&E tools to capture local experiences. 


While some scholars have identified the person-to-person ‘othering’ 
effects of power imbalances, Coultas discusses the ‘self-other’ relations in 
evidence-making involving person-to-artefact interdependencies through 
the M&E intervention. This insightful observation shows the power 
imbalances between GHPs and local partners, expressed through M&E 
artefacts. The evidence shows that the M&E artefacts wield power and 
represent the invisible presence of the GHPs from afar. In this process, 
Coultas successfully illustrates the conflict between the local implementer 
perspectives and the perspective of the artefacts. As a result, the dialogical 
approach reconciles the GHP logic at odds with local realities. Until this is 
corrected, pretentious reporting of M&E results leads to perverse self- 
silencing by local partners. 
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E Theoretical discussion 


This chapter employs a qualitative research design, focusing on a case 
study of Zimbabwe's Ministry of Health and Child Care. It relies on 
qualitative feedback from purposively selected M&E staff within the 
Ministry, supplemented by a critical analysis of four national M&E policy 
documents. The qualitative approach is chosen for its systematic and 
comprehensive data gathering, organisation, interpretation, analysis and 
communication, effectively addressing real-world concerns (Tracy 2013, 
p. 4). 


In examining the impact of GFATM’s partnership support for M&E in 
Zimbabwe's public health, the chapter draws upon context-specific and 
value-based insights perceived by M&E officers in the Ministry. It 
acknowledges the exploratory nature of the issues discussed, recognising 
that there is no single definitive answer to the questions posed by the 
problem under consideration. The research design aligns with a social 
constructivist ontology, subjective and value-based epistemology and the 
interpretivism paradigm, which asserts that perception originates from a 
self-reflexive subjective standpoint, with social and historical aspects 
preceding individual motivations and actions (Tracy 2013, p. 4). The chapter 
also argues that partnership for M&E is shaped by identity-forming 
discourses conducive to dialogic knowledge creation. It further recognises 
that the information generated may benefit some more than others. The 
chapter views partnership relationships between GFATM and the Ministry 
of Health as dynamic, requiring ongoing renegotiation, debate, and 
interpretation because of inherent power imbalances in real-life inter- 
organisational relations. 


Furthermore, the constructivist ontological perspective holds that social 
and political phenomena in partnerships are context- and time-specific. 
This implies that the meaning of policy ideas, like partnerships, is socially 
and politically constructed, contested, legitimised and sometimes 
strategically appropriated within existing power relations (Barnes 201], 
p. 58). Thus, the social constructivist or interpretivist epistemology in this 
chapter relies on the subjective creation of meaning from the literature 
reviewed, the researcher’s analysis of policy documents and the responses 
of key informants whose data inform the discussions. It underscores that 
the inquiry into the nature of knowledge and the validity of issues raised 
hinges on the beliefs and values of research participants, echoing current 
debates in critical literature regarding collaborative partnerships as a 
preferred model of public governance. 


Furthermore, language plays a fundamental role in understanding the 
social context in which local M&E officers experience the impact of GFATM 
support for M&E partnerships. Drawing from Van Dijk’s insights, the chapter 
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argues that CDA is a potent tool for identifying and resisting power (Van 
Dijk 2005, p. 352). Consequently, this chapter employs CDA to uncover 
instances of social power abuse, inequality and dominance within social 
institutions and to reveal the instrumental use of power and ideology in 
partnerships. This chapter highlights the fact that CDA can be used to 
perpetuate social inequality and domination, making it a valuable framework 
to illuminate ideological and power struggles often overlooked in the 
partnership between GFATM and the Ministry of Health and Child Care in 
Zimbabwe. Additionally, Khan et al. (2020, p. 149) illustrate how discourse 
implies a dialectical relationship in partnerships to create contextually and 
socially relevant knowledge and meaning. Therefore, the chapter’s adoption 
of CDA is a critical tool to uncover under-discussed and unintended issues 
within health partnerships in Zimbabwe. 


The chapter’s approach is further justified by the work of Michel Foucault, 
who views discourse as a comprehensive perspective that enables 
investigations into institutional oppressions and subjugations. Foucault’s 
poststructuralist school of thought and his examination of the relationship 
between discourse, thought and social practices underpin the framework 
used in this chapter. This approach provides insights into how the existing 
partnership between GFATM and the Ministry of Health benefits the former 
and its associates more than it benefits the latter and its local partners. 


E Evaluative inequalities in policy practices: 
The unintended effects of GFATM’s support 
to the Ministry’s monitoring and evaluation 
system in Zimbabwe 


The following section discusses the (un)intended effects of GFATM’s 
support to the Ministry of Health's M&E system, which includes the digital 
exclusion of local under-resourced partners, disruption of existing systems, 
clientelism, normalising parallel M&E systems, promoting brain drain, mute 
and perverse practices, partner contestations and ‘othering’, outcomes 
that perpetuate evaluative inequalities in the country. 


GHPs-supported monitoring and evaluation 
facilitating the digital exclusion 


The chapter indicates that the Ministry of Health and Child Care collaborates 
with the Global Fund and the president’s emergency plan for aids relief 
(PEPFAR) as a major funding partner to GFATM to facilitate coordinated 
and integrated collection of electronic patient-level and consolidated M&E 
data to enable clinical and management decisions. However, an analysis of 
the results reveals that this partnership had the unintended effects of 
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digitally excluding the Ministry’s other crucial departments and consortium 
local partners for the HIV, TB and malaria programmes. For example, the 
partnership for M&E exclusively supports electronic M&E systems like the 
electronic health records and the electronic patient monitoring system, 
which does not fully integrate the administrative and human resources 
indicators into the core reporting system. The system prioritises 
programme M&E performance indicators. Yet, administrative and human 
resource indicators have a predictive power to inform programme planning, 
monitoring and evaluation of the whole programme. Similarly, some local 
consortium partners for M&E are without the electronic system and hence 
find it difficult to contribute and retrieve information from the system. 


The exclusion of local partners and key Ministry of Health departments 
suggests unintended effects of GFATM’s interventions that perpetuate 
evaluative inequalities in the country. As a result, one argues that the 
current partnership does not fully satisfy the NPG theory principles of 
mutuality, organisational identity, trust building and shared decision- 
making as proposed by Osborne (2006, p. 381). The findings perhaps also 
suggest that the ideals of NPG theory have limits in promoting equity in 
local evaluation initiatives in unstable economic environments such as 
Zimbabwe. Scholars (Bopp, Harmon & Voida 2017, pp. 3611-3613) observed 
similar consequences of nonprofits’ and social enterprises’ data-driven 
work, leading to the erosion of local partner autonomy, data drift and data 
fragmentation contrary to the goals of productivity and empowerment. 


Thus, the GFATMs’ support for electronic health M&E systems in 
Zimbabwe excludes other less-resourced partners, thereby unintentionally 
perpetuating evaluative inequalities contrary to the mutuality, trust and 
reciprocity values that underpin partnerships as articulated in the NPG 
theory. This makes the theory insufficient to comprehensively explain 
partnerships involving parties that have different levels of resource 
endowments. As a result, there is a risk that the theory institutionalises the 
impossibility of equitable evaluations in Zimbabwe. 


Normalisation of parallel monitoring and 
evaluation systems 


The analysis of interview responses and documentary review of M&E 
policies reveals the (un)intended effect of normalisation of parallel GHP- 
supported M&E systems in Zimbabwe. The chapter reveals that GHPs 
finance parallel electronic M&E systems to facilitate quick turnaround 
reporting to donors. As a result, the Ministry coordinates many donor- 
specific M&E systems that have been normalised as part of the reporting 
system. Key informant interviewees concurred that parallel reporting is 
helpful as it provides opportunities for data triangulation. However, the 
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chapter highlights that while data triangulation is essential in M&E, 
sponsoring a parallel reporting system is neither a solution nor a data 
quality check intervention. 


As a result, the parallel data reporting systems do not consolidate and 
facilitate integrated information management for local decision-making 
and programme learning. Ultimately, the arrangement perpetuates 
evaluative inequalities in a fragile health system suffering from weak 
governance and the effects of economic sanctions since 2001. 


The normalisation of parallel systems contradicts the conventional view 
that parallel M&E systems are pathological to local M&E systems. The 
conventional view considers partnerships for M&E as progressive and 
functional (Craveiro & Dussault 2016, p. 482; Jain & Zorzi 2017, pp. S96-S98). 
This chapter reinforces this conventional view and further states that the 
existence of parallel systems implies parallel rather than shared beliefs, 
capacities and evidence of a lack of principled engagement. However, the 
chapter also highlights the need to clarify what constitutes parallel systems. 
The argument by one respondent hinges on the impossibility and 
unnecessary expectation of the ministry and local partners like Organization 
for Public Health, Interventions and Development (OPHID) trust to use one 
reporting system if their systems contribute to the same goal. The argument 
is plausible within the government-NGO partnership framework (Brinkerhoff 
2002, pp. 22-23), emphasising organisational identity and mutuality as key 
to partnerships. Thus, this chapter suggests the need for partner 
independence guided by national M&E goals. To this extent, parallel 
reporting systems are tolerated. 


However, considering the weak coordination and leadership roles from 
the Ministry of Health perspective, this chapter maintains that partner 
independence, as suggested in the government-NGO partnership 
framework and opinion, requires contextualisation as it proves unproductive 
in weak health systems. This is particularly concerning considering that the 
Ministry of Health receives over 90% of its budget from external partners, 
making the partnership uneven (PEPFAR COP 2020). While partnerships 
are not evaluated based on financial support alone, the study reveals that 
GHPs also supports administrative, human resources and computer 
equipment support to the Ministry. To that extent, one argues that the 
GFATM support for evaluative programmes in the public health system 
constitutes parallel systems to a significant degree. As a result, they 
perpetuate evaluative inequalities in the public health sector. 


Digital disruptions in monitoring and evaluation 


This chapter further reveals that while the GFATMs’ support to the 
Ministry of Health and Child Care intended to facilitate smooth 
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data-collection and reporting, the parallel electronic health systems 
lacked the appropriate infrastructure to support the continuous 
electronic data capturing and reporting. For example, the collaborative 
partnership for M&E has not provided the appropriate infrastructure to 
support the continuous electronic data-collection and reporting required 
to manage parallel data management processes for clinical staff, 
resulting in adverse health care system disruption. As a result, some key 
electronic processes that rely on electricity availability get disrupted 
during load shedding, necessitating the paper-based system’s 
maintenance. Thus, the GFATM’s intended goal of facilitating smooth 
data-collection and reporting has had the opposite effect, with public 
health clinicians soending more time consolidating and validating paper- 
based reports because of the system’s failure. 


While this observation does not directly link the unavailability of 
electricity to GHP support, it highlights the need for thorough feasibility 
of these interventions and priorities. As a result, disruption to data- 
collection, analysis and reporting leads to insufficient data availability for 
informed evaluative decisions. Yet, the GHP’s support for technology- 
driven discourses overlooks the practical challenges accompanying its 
use in environments with unsuitable infrastructure like Zimbabwe. 
Ultimately, the well-intended support for efficient electronic M&E systems 
becomes disruptive because of poor infrastructure to support the plan. 
The results include poor evaluative outcomes perpetuating exclusion 
from the global evaluation systems. 


However, this chapter acknowledges that digital disruptions are 
sometimes necessary to activate transformative evaluative change. In 
the current business discourse, ‘digital disruption’ has opportunities to 
improve health care systems (El Khatib et al. 2022, p. 564). Similarly, 
public administration scientists are experimenting with concepts such 
as digital governance, the Fourth Industrial Revolution (4IR), and public 
administration by algorithms (Bunasim 2020; Veale & Brass 2019). 
However, it is unclear if the digital disruptions in Zimbabwe's health care 
system derive as much value considering the need to attend to basic 
data-collection systems before investing in advanced virtual systems. 
The argument does not suggest antagonism to investments in technology 
to improve M&E systems. It draws attention to the need for thorough 
feasibility assessments to ensure the implementation of appropriate 
systems to avoid disruptions that contribute to evaluative inequalities 
because of a lack of quality data for decision-making. Where the 
infrastructure is unsuitable, digital disruptions contradict the spirit of 
partnerships as they fail to bring beneficiation to the system. 
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Contested nature of GHP-supported monitoring 
and evaluation 


This chapter further reveals the contested nature of GHP-supported M&E 
partnerships in Zimbabwe. This chapter illustrates concerns regarding 
using technology to circumvent health data security protocols involving 
some GHPs. They characterised these circumstances as sovereign invasion, 
which suggests subtle acknowledgement of contests contrary to the 
partnership principles of trust. Highlighting the sensitivities and importance 
of safeguarding local health data, one respondent referred to data as the 
‘new oil’. Drawing from CDA, the statement presupposes the reference to 
‘new oil’ as an illustration of the contested nature of extractive oil mining 
processes in many parts of the world. Likewise, it graphically depicts the 
‘extractive’ and disruptive nature of data-collection procedures by GHPs 
like the GFATM in Zimbabwe. 


The above discussion resonates with scholars’ similar observations 
(Gimbel et al. 2018, p. 80), referring to health data as the ‘new oil’. The 
context of the respondent’s answer is complaining about GHP activities, 
which signify contestations in the partnership. To this extent, the GHP’s 
support for M&E partnerships fuels mistrust contrary to its values of 
mutual engagements. Thus, the discourse of partnerships conceals the 
subtle contestations that prolong opportunities for frank and 
transformative conversations, thereby perpetuating evaluative inequalities 
in Zimbabwe. Partnerships driven by subtle confrontations concealed in 
partnerships delay opportunities to transform them to deliver locally 
relevant interventions. 


Patron-clientelism 


This chapter also highlights how GHPs like the GFATM unintentionally 
promote patron-client relations involving seconded champions in the 
Ministry of Health and Child Care. Key informant responses indicate a trend 
whereby senior staff on the chief operating level (c-suite) receive material 
incentives and salary support for their level of effort in GHPs-supported 
projects. Whereas GHP support with materials like vehicles, computers and 
finances facilitates the implementation of beneficial programmes in 
Zimbabwe, this chapter highlights how partnerships for M&E import 
practices are akin to patron-client relations. 


This chapter further observes that partnership terminologies such as 
‘incentives’ or ‘salary support’ conceal the intentions of GHPs as some of 
the interventions compromise the senior staff who find themselves 
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obligated to grant access to carefully framed demands for information 
from the GHPs. Thus, incentivising senior bureaucrats represents a soft 
power strategy that reduces resistance to information requests without 
recourse to expensive and openly contentious means. The discussion raises 
concerns as senior bureaucrats advance GHP’s interests, signifying missed 
opportunities to direct the GHPs to align their support to national health 
priorities. As a result, local evaluative needs receive less attention, as 
reflected in higher level-focused interventions at the quinary levels with 
limited support for primary level data needs. This chapter suggests a 
destabilisation effect in the national evaluative system as donor priorities 
take precedence. These challenges manifest in how GHPs decide evaluative 
terms of references and methodologies. 


The observations above resonate with findings by Mapitsa et al. (2019, 
p. 125), who assert that evaluations’ donors or commissioners define the 
choice of evaluations and its methods. This chapter further suggests that 
the soft power effects of material incentives potentially create conditions 
described by Alvesson and Spicer (2012, p. 1196) as functional stupidity or 
functional ignorance. The concept describes the making of questionable 
decisions, presumably out of ignorance. The process sometimes involves 
making the decisions that include ‘not to decide or act’ or decisions not 
ordinarily expected from patriotic and informed leaders in the C-suite. As a 
result, the questionable decisions have a direct link to the material support 
to the staff that amounts to patron-client relations. Ultimately, policy 
decisions do not reflect the evaluative priorities for the country but external 
donors and commissioners. 


However, patron-client relations are not as linear and unidirectional as 
suggested in the preceding discussions. The traditional view of patron- 
client relations assumes capture without acknowledging the possibility of 
agentive reflexivity by the local bureaucrats, who may instrumentally 
appropriate the GHP resources for the greater good as an obfuscation and 
extraversion strategy. Patterson (2018, p. 16) asserts that because state 
elites seek rents, they look to benefit from donor health resources. Similarly, 
Herrick (2018, p. 1) and Okeke (2018, p. 1) remind us of the precarity and 
temporality of partnerships involving GHPs in collaborative partnerships 
with perceived weaker governments. Their studies reveal that the buck 
stops with the government in all collaborative governance partnerships. 
Thus, even in the most unstable health economies, government bureaucrats 
wield some power over well-resourced partners from developed countries. 
However, whichever way one views these relations, they expose the limits 
of partnership discourse as they fail the test of mutuality and trust. As a 
result, they delay opportunities to narrow the evaluative gaps at local 
health systems level. 


194 


Chapter 9 


GHPs and monitoring and evaluation brain drain 
in the health sector 


The chapter also reveals GHPs’ contribution to the internal and external 
brain drain of key ministry M&E staff as they join local and international 
NGOs. Through analysing key informant responses, this chapter finds that 
the GHP capacity-building support for local and international M&E training 
programmes and local skills-building initiatives such as HIV population- 
based modelling, HIV estimates and big data analytics has had (un) 
intended effects of preparing the staff for lucrative local and international 
NGOs posts. Thus, the study observed that M&E staff in the Ministry 
appropriate these opportunities for personal skills and professional 
development to launch lucrative career opportunities for better rewarding 
jobs in the NGO sector. When this happens, the Ministry lacks adequate 
staff to coordinate evaluative programmes. As a result, some of the previous 
ministry staff were left to coordinate the programmes from across the 
partnership aisle as partners. Thus, this chapter highlights the disruptive 
effects of the partnerships for M&E, resulting in widening evaluative 
inequalities through dependence on donor resources. The observations do 
not suggest capacity-building for ministry staff as an_ ill-informed 
intervention but highlight the need for reflection by both the Ministry and 
the GHPs like GFATM on the unanticipated effects of these partnerships. 
Therefore, these observations call for reflection in the partnership discourse 
that presents capacity-building as a (un)disputed good intervention 
(Herrick & Brooks 2018, p. 528). 


The GHPs supporting capacity-building initiatives aim to strengthen 
the systems, not create opportunities for individual career growth. To this 
extent, the unintended effects of GHP support perpetuate evaluative 
inequalities through skills flight from the Ministry. While Shukla (2013, p. 68) 
would classify the personal staff opportunities as unanticipated positive 
effects of GHP support for M&E, this chapter draws attention to its adverse 
effects on the evaluative system. 


Drawing from the critical-governmentality literature, the discussion on 
brain drain invokes the current conceptualisation by Lilja and Baaz (2022, 
p. 190) regarding an increasingly new form of governance in which human 
resource subjects’ desires for career professional improvement and 
creativity are extorted to control and profit from them. The concept that 
the scholars call ‘arte-politics’ is a form of governance technology that 
seeks to regulate individual behaviour in terms of self-realisation and the 
distribution of ‘freedom’ to control the labour situation of the employees. 
The demand for capacity-building on technical M&E skills suggests the 
influence of arte-politics in Zimbabwe's public health M&E system. At the 
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same time, distributed freedom and self-realisation represent some form of 
neoliberal exploitation. Despite being beneficial to individual careers, it is 
detrimental to the partnership goals for local health systems. Ultimately, it 
perpetuates evaluative inequalities through the attractive power of arte- 
politics manifesting as brain drain. 


Promoting mute and perverse practices 


This chapter also further reveals those partnerships for M&E with GHPs like 
GFATM manifest pathological and harmful practices (perverse) to local 
health systems goals. For example, this chapter highlights concern over the 
high targets GHPs like the GFATM set for the government and its local 
partners to achieve in programmes like medical male circumcision to 
prevent HIV. The GHP support links the achievement of targets to further 
funding under the results-based funding framework. This chapter asserts 
that some local partners connive with health facility staff to report falsified 
results to the donors to facilitate the next financial disbursement. 


Furthermore, this chapter reveals that the attractive desire to continue 
receiving foreign aid for evaluative programmes silences the voice of local 
partners (mutes) and causes blind following on crucial policy issues. Unlike 
the patron-client relations focused on an individual level, mute and perverse 
practices are at the policy level, where the government deliberately allows 
certain processes (not) to happen if they can facilitate further funding. For 
example, the government of Zimbabwe embraced public health programmes 
targeting key populations like males having sex with males, departing from 
its previous critical stance on the issue (Zimbabwe - OHCHR UPR 
submissions 2022). Although the government’s policy position on this issue 
remains unclear, the current major funded HIV and TB programmes are 
targeting the key populations. The plausible explanation for this situation is 
the government’s desire to access the HIV and TB programmes that 
exclusively target the key populations. The observation, therefore, seeks to 
highlight the possibility of missed opportunities to respond comprehensively 
to the genuine public health demands for key populations. 


The chapter raises fears that implementing programmes to attract more 
funds on the part of the government and fulfil donor requirements on the 
part of GHPs does not facilitate comprehensive service provision based on 
evidence. Whichever way one views the politics of this situation, the country 
misses an opportunity to develop comprehensive evaluative frameworks to 
identify and respond to the ever-increasing needs of key populations. Thus, 
the lack of mutuality and trust between the partners perpetuates practices 
that delay opportunities to design and implement comprehensive evaluation 
frameworks for effective interventions. The situation promotes possibilities 
for falsification of information. Therefore, GHPs and the government should 
engage in open conversations consistent with the principles of partnerships. 
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Scholars such as Makuwira (2018, pp. 425-430) and Storeng et al. (2019, 
pp. 555-564) share similar reflections in which governments enter into 
partnerships with GHPs to facilitate further funding in Malawi and South 
Sudan. The experiences in these countries involved less-than-admirable 
strategies, such as falsifying data to compensate for missed collaborative 
partnership targets. The practices show a lack of open and transparent 
communication based on trust and mutual interest in the functioning 
of GHPs. 


‘Othering’ and conceptual boundaries 


The final discussion point in this chapter reveals that the partnerships for 
M&E tend to create conceptual boundaries and the unintended effect of 
‘othering’ among the ministry staff. Othering derives from the notion that 
an ‘us’ and a ‘them’ oppose one another. A false distinction is drawn 
between the ‘us’ and the ‘them’ to subjugate one group to another (Strand 
2018, p. 13). This chapter reveals that GHP’s support for M&E in Zimbabwe 
has created demand and necessitates higher qualifications for M&E 
positions as a prerequisite for employment. As a result, the M&E role has 
attained a special status for ‘other’ staff with a background and qualifications 
such as statistics and computer science, biostatistics and new degrees in 
M&E. This chapter observes that staff without M&E qualifications are 
referred to as ‘other’ staff, and those with qualifications are referred to as 
‘us’. As a result, the M&E positions occupy an exclusive status in the 
development arena in Zimbabwe. 


However, the unintended binaries of ‘us’ and ‘others’ have not been 
problematised enough despite their potential effects in perpetuating 
evaluative inequalities. For example, it creates conceptual boundaries 
between expatriate (us) and local (other) M&E staff. Similarly, the local 
M&E staff (us) look down upon ‘other’ programme and support staff 
members. As a result, conceptual boundaries around M&E have had the 
unintended effects of reinforcing staff divisions contrary to the principles 
of partnerships. 


The chapter’s arguments about the ‘othering’ effects of GHP between 
supported M&E staff (us) and unsupported staff (others) are consistent 
with findings by Peters (2016, p. 8) and Mueller-Hirth (2012, p. 663). They 
identify adverse effects of professional M&E boundaries involving 
expatriates, local M&E and programme staff in Angola and South Africa. 
This chapter, therefore, concludes that GHP support for M&E has created 
conceptual boundaries inconsistent with the key tenets of mutuality and 
trust among project staff. As a result, the partnerships are contested spaces 
rather than platforms for facilitating knowledge diffusion for innovative 
evaluative initiatives in the country. The ‘othering’ practices perpetuate 
power imbalances that resemble the paternalistic behaviour of its funders 
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in the collaborative networks for M&E in Zimbabwe. These practices are 
counterproductive and contrary to the core partnership values of trust and 
mutual interest. 


E Conclusion 


In conclusion, this chapter draws on the Zimbabwean Ministry of Health 
and Child Care Zimbabwe's case study to demonstrate the Cun)intentional 
effects of GHP support in widening rather than bridging the evaluative 
inequalities in the country. This chapter has highlighted the practical, 
unintended negative effects of collaborative health partnerships, providing 
new perspectives that contrast Eurocentric, instrumental views about 
collaborative partnerships. The anticipated benefits of partnerships to 
promote democracy, shared decision-making (plurality), equity and 
improved public service delivery by diffusing political power from the 
sovereign state to its private and civil society partners have not been 
realised. Furthermore, the anticipated improvements in evaluative processes 
because of the comparative involvement of GHPs have remained elusive in 
Zimbabwe. Thus, the anticipated merits and virtues of the partnership have 
not translated into meaningful progress in the quest for strengthening M&E 
systems in Zimbabwe. As a result, the well-known benefits of collaborative 
partnerships are problematised in this chapter, arguing that its vaunted 
benefits of mutuality, trust and reciprocity do not exist in practice. This 
makes partnership discourse in its current form the greatest threat to 
efforts to facilitate evaluative equity in Zimbabwe. 
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Why emphasis on 
randomised controlled 
trials is a flawed approach 
to evaluation for a more 
equitable society 


Sean M Muller 

Institute for Advanced Study, 

Faculty of Humanities, University of Johannesburg, 
Johannesburg, South Africa 


E introduction 


On the face of it, evaluation has a very practical concern: to establish the 
effects of a programme and how the programme played out relative to 
what was intended. In African countries, the programmes in question are 
also typically concerned with the very practical matters of addressing 
components of poverty, inequality and ‘underdevelopment’ - or their 
ostensible causes, such as weak state capacity, low levels of education, 
credit constraints, gender biases, limited technological transfer or 
technological know-how and so forth. Yet all important aspects of 
programmes and evaluations are subject to a range of seemingly more 
abstract considerations and concerns. For programmes, these range from 
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the question of which challenges should be prioritised and chosen for 
programmatic interventions to who makes that choice, where and at what 
scale the programme is implemented, how a notional intervention takes 
place in practice, etc. Where there is a connection between evaluation and 
programme, many of these concerns will carry over - indeed, it is good 
practice when evaluation is going to take place for issues that arise at the 
evaluation stage to be considered at the programme selection and design 
stage.” 


Evaluations themselves face an additional set of concerns that are more 
distinctly methodological and philosophical in nature. Notably, seeking to 
ascertain ‘the effect’ of a programme means potentially grappling with the 
challenges of inferring the causal effect of one social phenomenon on 
others. That requires some method for observing the relevant phenomena, 
disentangling them from others, and potentially measuring them 
quantitatively in order to infer the magnitude of effects. The large literature 
on these and related topics across many disciplines, from epidemiology to 
economics, partially reflects the difficulties in credibly addressing them. 


Yet the decisions taken in these respects may be flawed in a variety of 
ways that can negatively impact the primary objective of building more 
just and equitable societies. This chapter seeks to illustrate this using 
the example of one particular approach to evaluation that has become 
particularly influential in the field of development economics - the 
use of randomised controlled trials (RCTs). It begins by providing 
some background and context to the RCT approach. | then discuss two 
crucial flaws in that approach, which | suggest fatally compromise the 
ostensible aims of evidence-based policy (EBP) in development and 
therefore the achievement of just and equitable societies. The first of these 
is that particular worldviews, biases and prejudices enter into the evaluation 
process using RCTs at the very outset yet go entirely unacknowledged 
and instead are concealed behind a veneer of ‘scientific evidence’. These 
impose a limited and distorted understanding of inequality and its 
generative mechanisms at the very outset of the research process. The 
second is that the way in which the approach informs policy decisions 
reflects dubious and harmful epistemic hierarchies that inappropriately 
valourise certain kinds of knowledge and certain kinds of persons. That 
compounds the harms of the first problem and converts them into practical 
ones by influencing policy decisions. The pursuit of just and equitable 
societies can thereby be diverted for long periods of time or even turned 
in a direction that leads to diametrically opposite outcomes. Evaluators 
who adopt such approaches are made complicit. The concluding section 


17. Whether and when formal evaluation is required or desirable is a separate matter. 
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briefly discusses how the African Evaluation Database (AfrED) may be 
used to examine the influence and manifestation of some of the concerns 
expressed in preceding sections and how evaluators may limit or mitigate 
their role in such harms. 


E Background: From causal inference to 
epistemic and practical harms 


Beyond the more immediate methodological questions that face 
programme evaluators are philosophical ones about what causality is 
(Cartwright 1989; Mackie 1974; Woodward 2003), when and whether causal 
effects can be identified, whether it makes sense to think of such effects in 
isolation from mechanisms (Cartwright 2007a; Clarke et al. 2014), whether 
causal effects can be identified algorithmically using causal graphs and 
observational data (Spirtes, Glymour & Scheines 1993), whether they can 
be extrapolated (Muller 2021a) and so on. 


One may also question whether ascertaining the success of a programme 
through identifying a causal effect is the right approach; one could instead, 
for example, ask the intended beneficiaries of the programme whether 
they are satisfied with it. Proponents of EBP notionally endorse the idea 
that evidence should feed into democratic processes - albeit in a very 
particular way mediated by researchers and technocrats like themselves. 
So, from that perspective, the soliciting of opinions from members of the 
public directly as a way of assessing the merits of a programme is defensible. 
In fact, strictly speaking, the intellectual commitments of mainstream 
economists typically prohibit them from making policy recommendations 
without knowing what the preferences of the relevant population are - 
which they typically do not. 


The supposed ‘credibility revolution’ 
in evaluation 


While such matters receive recognition in places within the evaluation 
literature, they are largely put aside when actually planning interventions 
or designing and conducting evaluations. Given the status quo in which 
evaluations are typically expected to provide some kind of confident or 
definitive assessment (provided there were no hindrances in the process), 
this is somewhat inevitable. The concern of the present chapter is with the 
emergence in recent decades of an approach to evaluation that purports to 
overcome such challenges or render them redundant, thereby justifying its 
claim to being the ‘gold standard’ and heralding a ‘credibility revolution’ 
(Angrist & Pischke 2010). 
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The approach in question has been summarised in many publications and 
‘how to’ guides by proponents (Banerjee & Duflo 2009, 2017; Gertler et al. 
2016; Glennerster & Takavarasha 2013). It typically starts with researchers 
identifying a policy question of interest and then narrowing that down to a 
sub-question that can be analysed using a ‘field experiment’ in which a 
social or economic intervention is assigned to a relevant group of ‘subjects’ 
randomly. The purpose of randomisation is to eliminate the influence of 
confounding factors and thereby be able to make a claim about the causal 
effect of the intervention while side-stepping the statistical or econometric 
assumptions and additional data needed in the absence of randomisation.” 
In some instances, researchers utilise results from a single such evaluation 
to make policy recommendations, either in the original context or in other 
contexts. In other instances, the apparent causal effect on the outcome of 
interest may be compared to that from evaluations of other interventions 
or contribute to synthesised findings from meta-analyses of evaluations. 
The broader practical objective is to improve or advance economic and 
societal ‘development’. 


Critical perspectives and epistemic harms 


While similar to - and partly inspired by - the approach taken in medicine 
to pharmaceutical trials (Cartwright 2007b), the approach and its influence 
have been driven by a small group of mainstream economists largely based 
inasmall number of academic departments in the United States. Moreover, 
it has been far removed from the typical, or stereotypical, disinterested 
academic enterprise, with the recipients of the Nobel Memorial Prize (Nobel 
Media AB 2019) all having been involved in founding institutions that 
have actively sought to persuade funders, governments and civil society 
organisations to adopt the approach well before it obtained widespread 
support from academic peers and with relatively little regard for concerns 
that were raised by other scholars (Bédécarrats, Guérin & Roubaud 2019). 
Moreover, while there has been an effort to frame the endeavour and its 
proponents as ‘modest’ by virtue of focusing on small rather than ‘grand’ 
interventions (Banerjee & Duflo 2011), close scrutiny does not support that 
conclusion (Bardhan 2013; Bédécarrats et al. 2019; Harrison 2011, 2013; 
Muller 2021a). 


The geographical and academic location of the core of the ‘randomista’ 
(Ravallion 2009) movement reflects, | suggest, two overlapping hierarchies. 
The first is the global epistemic hierarchy of knowledge in which scholars 
and others based in the Global North are deemed to be inherently more 


18. Statistically the idea is that the influence of confounding factors is either eliminated or balanced across 
the treatment and control groups. Even the statistical efficacy of this is not uncontroversial but those 
technical issues are outside the scope of the present chapter. 
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insightful and credible than those in the Global South, even about matters 
pertaining to countries and people in the South, resulting in concomitantly 
greater influence in various domains.’? This phenomenon has been written 
about extensively, and the associated way of thinking also exists in the 
Global South itself, where such ways of thinking have been imposed or 
transferred (Hountondji 1990; Ndlovu-Gatsheni 2018). The second is the 
extreme hierarchical structure within mainstream, neoclassical economics 
itself in which it is taken as given that graduates and academics of top- 
ranked departments in the United States (and a few in Europe) and those 
who publish in the journals of those departments are inherently superior to 
other economists - including African economists on issues pertaining to 
African countries (Chelwa 2021; Fourcade, Ollion & Algan 2015; Heckman & 
Moktan 2020; Muller 2021b).2° 


This background is important to recount here because many practitioners 
(and even scholars) of evaluation are unaware of such details and nuances, 
focused as they are on practical and immediate methodological 
considerations. The subsequent sections will develop concerns that partly 
emerge from this background. Firstly, contrary to the framing of RCTs as 
‘scientific’ and value-free ex-ante, they in fact reflect the worldview and 
values of those conceiving them. Secondly, the explicit and implicit epistemic 
hierarchies on which the randomised approach is premised are dubious and 
presently do not withstand critical scrutiny. Relatedly, the manner in which 
RCTs have been, and continue to be, deployed as a form of evaluation 
compounds asymmetries between the Global North and South that can be 
considered present-day extensions of colonial power relations in the domain 
of knowledge (Hountondji 1990; Ndlovu-Gatsheni 2021). 


Each of these issues can be considered a ‘harm’ in and of itself. For 
example, there is a large and rapidly growing literature on what has been 
called ‘epistemic injustice’ (Fricker 2007; Koskinen & Rolin 2019), but the 
focus here is on the more direct practical implications. The thrust is not just 
that randomised evaluations may fall short of achieving their objectives 
and claimed accomplishments - of providing evidence that facilitates social 
and economic development. The problems of social and economic 
development are among the hardest of those that exist, so merely falling 


19. Recently some scholars, typically in the Global North, have taken to expressing disquiet on social media 
about, or disparaging, the North-South distinction. Certainly, like any other binary, it is somewhat crude. 
However, such criticisms or concerns have little traction when utilised by scholars in the Global South as 
a distinction premised on solidarity or commonality among colonised or otherwise oppressed countries 
and it is in that sense that the terms are used in this chapter. The problems with the ways in which certain 
scholars in the Global North use these terms, or indeed others relating to ‘sub-Saharan Africa’, ‘the Third 
World’, ‘developing countries’ and so forth is a matter for separate analysis. 


20. Those interested in further detail can consult any one of a number of ‘economics job market guides’ 
published in North America in which such assumptions are often stated in passing or are implicit in the 
guidance provided. 
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short in addressing or ‘solving’ those would be little basis for criticism. By 
selecting hypotheses based primarily on the worldview of those based in 
the Global North institutions, utilising a method that favours non-structural 
interventions, producing findings that are not generalisable by the standards 
of the same paradigm it is based on, and leveraging the asymmetry of 
power relative to Global South governments and other local institutions, 
there is a strong basis for believing that these evaluations - individually and 
especially collectively - are and will be harmful and counterproductive 
(Muller forthcoming). And that ought to be of great concern to practitioners 
sincerely committed to the positive societal role that evaluations are 
intended to play. 


E The value-ladenness of randomised 
evaluations?! 


Educational interventions: Value for money and 
generalisability 


There is no strictly consistent way in which RCTs are used within the 
evaluation process, but there are some examples from proponents of 
these methods that may be used as exemplars. Consider in particular 
the paper in Science by Kremer, Brannen and Glennerster (2013) on ‘The 
Challenge of Education and Learning in the Developing World’. The core 
of the paper is represented by a table showing the estimated causal 
effects of different experimental interventions on test scores, with 
the latter normalised so as to be comparable across contexts (Kremer 
et al. 2013, p. 298). Accompanying these quantitative estimates are 
corresponding estimates of the costs of implementing the policies. Those 
two quantitative estimates are combined in a simple cost-benefit analysis. 
The logic being that policy-makers should select the intervention that is 
likely to yield the highest benefit (percentage increase in test scores) 
relative to cost. Moreover, normative or subjective factors, like trading off 
benefits for one part of a population against another, enter this final stage 
after the ‘scientific’ and ‘objective’ analysis has been conducted. On the 
face of it this indeed seems very ‘scientific’ and the publication of such 
papers by economists in what were previously general interest, highly 
prestigious natural science journals, including Nature, gives a substantial 
boost to economists’ attempts to frame themselves as scientists.?? 


21. This section draws primarily on Muller (2022). 


22. Elsewhere Muller (2021a) explains why mainstream economics might be better thought of as a ‘facsimile 
science’ but that argument would take us too far afield here. 
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That appearance is, however, extremely misleading. As already mentioned, 
there are a wide range of technical concerns that could be raised in relation 
to this approach. Comparability across contexts requires strong assumptions 
about interacting factors and causal mechanisms, yet neither issue is given 
any serious attention in the paper. Muller (2014) demonstrates that, within 
the limitations of the data and methods, in an influential, high-quality 
randomised evaluation of class size, the causal effects estimated were likely 
dependent on unobserved teacher quality.’ As teacher quality would vary 
across contexts, this would mean that the effects could not be responsibly, 
or ‘scientifically’, generalised without at the least information on teacher 
quality that is comparable across contexts - something that rarely exists. 
Yet other highly influential contributions have sought to make such 
comparisons of class size effects. 


Unacknowledged subjectivity and bias 


Putting such potentially fatal (Muller 2020) technical limitations aside, 
another crucial implicit omission relates to the entry of normative 
(subjective) factors within the processes that lead to a summary table like 
that found in Kremer et al. (2013). To understand the problem, one first 
needs to recognise the basic point that researcher judgement always 
narrows down the set of possible questions for a variety of reasons. Firstly, 
in relation to what topic is studied: education rather than health, educational 
achievement rather than educational experience, and so forth. Secondly, in 
relation to how that is studied: qualitative or quantitative methods, 
randomised trials rather than structural econometric models, etc. 


In economics, concerns had already been raised about the degree to 
which RCTs may constitute an instance of ‘looking for keys under the 
streetlight’; in other words, looking for a solution to a problem only where 
one is able to see, regardless of the likelihood that it is there. Selecting 
research questions merely because they can be answered by an RCT ora 
cleverly selected ‘natural experiment’ (Angrist & Krueger 2001; Meyer 1995) 
arguably amounts to this. One would need some kind of much deeper 
argument, connecting the method of discovery with the importance of 
what is discovered, to believe that this approach will reveal the truths that 
are most important to society. So, the problem stands even if one accepts 
the ostensible epistemic merits of these approaches (namely, that they are 
the best available for identifying causal relationships and effects). 


There is a further problem, however, that has been neglected or omitted 
in the extant literature on RCTs and other derivative literature: the normative 


23. The generic, technical problem is explained in Muller (2015). 
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(subjective) views of researchers conducting RCTs also narrow down the 
set of potential research questions (Muller 2022). The education example 
illustrates this well. Kremer et al.’s (2013) seemingly long list of potential 
interventions - 30 experimental evaluations tabulated from different 
developing countries - includes only three that constitute a large increase 
in resource allocation to schools (or the schooling system if they were to be 
expanded). These relate to decreasing class size. For example, to decrease 
class size from 80 to 40, one would need to approximately double the 
number of teachers if keeping other factors constant, which in turn would 
require a large increase in resource allocation as personnel budgets are the 
largest component of educational expenditure in all countries. The authors 
note that the two experiments in India and Kenya ‘did not significantly 
improve test scores’ and take this as evidence that resource increases in 
general ‘without changing pedagogy or accountability’ are ineffective. 


The reader might notice how remarkable it is that an evidence-based 
approach takes two experiments tangentially related to a question as 
sufficient evidence to adopt a strong stance on such a major issue (resource 
adequacy). As discussed by Muller (forthcoming), that dubious logic has 
been imitated by South African researchers in assessments of available, 
evidence (Van der Berg et al. 2016). The more immediate point is that there 
is no ex-ante, wholly objective basis for any particular composition of the 
types of interventions tested. So, to some degree, the composition reflects 
researchers’ prior views about what is likely to be worth trying, combined 
with the available resources and opportunities for conducting different 
kinds of experiments. In that way, we can infer that the community of 
researchers conducting such experimental evaluations did not, for instance, 
consider that increasing teacher salaries or otherwise seeking to improve 
teacher working conditions had any merit. It is important to restate the 
problem here. The issue is not, first and foremost, whether there is a debate 
to be had about alternative policies such as improving teachers’ working 
conditions. The point is more fundamental: that ‘randomista’ advocates of 
randomised experiments dismiss the notion that reliable knowledge for 
policy can exist unless it is premised on such evaluations, yet in their choice 
of what to evaluate they have assumed that certain kinds of interventions 
are simply not worth testing - resulting in a contradiction. In engaging in 
such subtly contradictory behaviour the result is to maintain a claim to 
objective, evidence-based, quasi-scientific approach to policy while 
smuggling in - wittingly or unwittingly - a host of prior beliefs, biases and 
the like that are unsubstantiated. The approach taken and advocated by 
Kremer et al. (2013), Banerjee and Duflo (2009, 2011) and others is one in 
which normative factors enter into the policy process by narrowing down 
the kinds of research questions that are asked and corresponding 
experiments that are conducted (Muller 2022), 


206 


Chapter 10 


While experiments related to education were the most cited in the Nobel 
award for the use of randomised trials in development (Nobel Media AB 
2019), the same issues can be identified in other research areas. Notably, 
the overly hasty dismissal of resource increases and large-scale structural 
changes as potential solutions to problems in developing countries is not 
limited to the education sector - it is characteristic of most of the RCT 
literature. Possible explanations for that are somewhat speculative but can 
partly be traced to a rejection of ‘grand theories of development’ in favour 
of micro-level interventions (Banerjee 2007, 2012; Banerjee & Duflo 2011). 
The latter orientation was based on the observation that approaches based 
on such grand theories appeared not to have worked. Certainly, there is 
little doubt that the majority of so-called developing countries, with the 
much-analysed exception of the ‘East Asian tigers’, did not experience 
rapid economic growth or socio-economic development under such 
approaches, and many ran into serious economic difficulties that continue 
to the present day. But diagnosing the reasons for such failures and whether 
they reflect a broader failure of such structurally-oriented approaches is a 
completely different matter. The jump to asserting that micro-level 
interventions will provide an alternative, more modest ladder towards 
socio-economic development arguably exemplifies what one notable 
scholar of development called ‘the pessimism of the diagnosis and the 
optimism of the prescription’ (Mkandawire 2001). 


Despite the role of subjectivity in the selection of research questions 
having been noted many decades ago (cf. Myrdal 1970), these problems 
have been entirely omitted from the RCT literature.** The role of values, 
including biases and prejudices, in designing randomised evaluations is 
neither recognised nor dealt with in any substantive way in the economics 
literature. In principle, the discipline endorses a strict separation between 
subjective and objective considerations in economic analysis, typically 
phrased as being between ‘normative’ and ‘positive’ analysis (Badiei & 
Grivaux 2022), but in practice, the separation is materially violated. These 
problems spill over into the evaluation literature, where, if anything, they 
are compounded by the typically narrow focus on single interventions. 
Evaluation and its conclusions, or prescriptions, therefore inherit the 
problems of the randomista approach. 


E Dubious epistemic hierarchies 


At the heart of much of the debate about RCTs in the quantitative social 
science literature has been their purported status as a ‘gold standard’ for 


24. The issue has also been omitted from the recent literature on the role of values in science. A discussion 
of, and explanation for, that is left for discussion elsewhere. 
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causal identification and evaluation of interventions (Cartwright 2007b; 
Deaton & Cartwright 2018) - somewhat mirroring earlier debates in the 
medical literature (Concato, Shah & Horwitz 2000). The use of the term 
itself has waned in the face of criticism, but the substantive commitment of 
many researchers and scholars to the idea that experimental evidence is 
inherently superior to other forms of evidence finds expression in statements 
about ‘rigour’ and ‘robustness’ of research. In essence, the debate has two 
strands. The first disputes whether RCTs should be at the top of the 
evidence hierarchy, while the second disputes whether there should be a 
generic evidence hierarchy at all. As noted earlier, if one believes that 
evaluation is only about causal identification, and if one believes that RCTs 
are the best (most reliable) way of obtaining such identification, then it 
follows that RCTs should be represented as superior to other forms of 
evidence. Where this has manifested most explicitly is in meta-analyses, of 
which economists have in fact been late adopters, where studies that are 
not based on randomised evaluations are often simply excluded from 
consideration (Abimbola 2020). 


The appeal of an evidence hierarchy is quite clear. For the researcher, it 
provides a simple way of selecting a method of analysis: consider the 
feasibility of the method at the top and work one’s way down. For the 
policy-maker, it serves a similar purpose: when considering evidence they 
need only start with evidence based on methods at the top of the hierarchy, 
and then only consider other evidence if the former is not available. In this 
sense, evidence hierarchies are an ‘easy sell’ - something proponents of 
RCTs have been keen to emphasise about their approach in general 
(Labrousse 2021) and that has been very successful in the ‘evaluation 
market’ (Manski 2011, 2013; Picciotto 2021). Unfortunately, as many have 
argued, the simple appeal of hierarchies does not appear to correspond to 
the complex realities of the phenomena being studied. In this case, evidence 
hierarchies create new problems rather than resolving existing ones. 


As with the basic claim that RCTs can be used to identify causal effects 
that are useful for policy, there are many technical challenges that can be 
posed to the more specific claim that RCTs are superior to other forms of 
evidence - even just for identifying causal relationships and effects.?° 
However, there are greater questions about evidence hierarchies that have 
RCTs at the top. Crucially, such approaches entirely devalue local knowledge 
and render it replaceable by, and fundamentally inferior to, externally- 
initiated experiments. By ‘local knowledge’, here, | mean the knowledge 
about particular phenomena or processes that is held by those directly 


25. See, for instance, the literature associated with Heckman and co-authors (Heckman 2001; Heckman & 
Smith 1995; Heckman & Vytlacil 2005) or the literature associated with Manski’s work (Manski, 1993, 201], 
2013). 
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involved, those in relevant (typically national) institutions that may be 
responsible for problems or solutions and broader groups such as policy- 
makers and the citizenry in general. To sharpen the point a little: making 
RCTs ‘the gold standard’ - or claiming they produce the most ‘rigorous’ 
form of evidence - means that we ought, unquestioningly, to value evidence 
from an education experiment conceived, funded, directed and analysed 
by a North American researcher over the opinions or insights of acompetent 
and committed teacher working within the education system where 
improvements are being sought. For economists, including those who are 
not RCT proselytisers, this is a straightforward step because qualitative 
analysis has been explicitly disparaged and discarded in modern 
mainstream, neoclassical economics. But the basis for that is weak and 
there is little reason why those actually concerned with societal problems, 
rather than conforming to what is arguably a facsimile science (Muller 
2021a), should do the same. 


If one declines to simply accept the wholesale discarding of non- 
quantitative, non-experimental knowledge, then the proposed evidence 
hierarchy raises a series of broader epistemological questions. How is it 
that we as human beings can best learn about the dynamics of the societal 
systems we are in and construct? If we place weight on knowledge gained 
from participating in these systems, to whose views and insights do we 
give more or less weight? Moreover, how can we best determine what 
actions to take in order to achieve a range of desired outcomes? And one 
might also ask to what extent are these - understanding and successfully 
intervening in societal systems - even achievable. Neoclassical economists 
have been operating for almost a century as if these large questions have 
been answered and the randomista approach should be understood within 
that context. The question of whose knowledge is given weight is, in 
contrast, the subject of a number of pieces of literature in other disciplines 
like sociology, anthropology and philosophy. 


Indigenous and local knowledge 


One particular strand of that literature is concerned with ‘indigenous 
knowledge’ (Lee & Evans 2022; Smith 2012; Wiredu 1996) and ‘indigenous 
knowledge systems’ (IKSs) and emphasises the role of colonial histories 
in suppressing, devaluing and appropriating the knowledge of colonised 
peoples. And some recent scholarly contributions (Koskinen & Rolin 
2019) examine how remedying this may be epistemically beneficial - in 
other words, improve understanding of particular societal questions. 
Much of the harm done by colonial approaches occurred under the 
rhetoric of ‘science’ and that has led to a tendency of some authors to 
contrapose science and indigenous knowledge. In my view, this is 
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unnecessary and counterproductive. As argued elsewhere, it can be 
helpful to recall that the same colonising societies also had their own 
‘indigenous knowledge systems’ that were also discarded and 
disregarded, although often not in as brutal or exploitative a fashion 
(Muller 2021c). Recognising that helps to some degree in seeing that the 
problem was not in the notion of ‘science’ per se but rather in the manner 
in which it was implemented and coupled to other agendas. Thus, 
science may well show that some indigenous knowledge is ‘incorrect’ 
while some indigenous knowledge may reveal that certain scientific 
understandings are or have been incorrect. The historical record provides 
evidence of both types of occurrences. And if one sees these sets of 
knowledge as having been obtained through different ways of 
understanding the world, then there is no sense in which one is forced 
to choose between them. 


All of this is very important for understanding what the adoption of a 
randomista evidence hierarchy in the evaluation context really implies for 
African countries, governments and peoples. It reproduces, in effect, the 
same kinds of epistemic and associated harms as the colonial and post- 
colonial eras (Hountondji 1990) under an analogous rhetoric of being ‘more 
science-like’ (Njoya 2020). Once again external actors (Panin 2020), with 
little direct or temporally extensive experience of the societies in question, 
design interventions on the basis that the evidence they have is superior to 
whatever else might exist or might have existed prior to their ‘arrival’ on the 
scene. When pressed they may resort to gesturing at their local functionaries 
who run the experiments for them, or they may deploy narratives about 
their travels among the local peoples.”® 


Ethical concerns 


One may then further link this to ethical questions pertaining to research. 
An increasing number of scholars have examined a range of ethical 
concerns relating to RCTs and found the approach, in general, to be 
compromised or, at the very least, in need of serious improvement 
(Abramowicz & Szafarz 2021; Baele 2013; Hoffmann 2020; Ziliak & Teather- 
Posadas 2016). In addition, more serious issues have arisen in relation to 
individual randomised interventions - involving everything from cutting 
off water access to exposing poor people to the proselytisation of 
missionaries (Muller forthcoming). Despite the controversies, such 


26. It is remarkable how durable the ‘Western researcher with native peoples’ photograph is over the 
space of centuries, despite fairly widespread awareness from the late 20th century of the offensiveness of 
such tropes and what they imply. Anecdotally, the only presentation this author attended at the University 
of Oxford by Esther Duflo, one of the three Nobel Memorial Prize winners, contained such photos of Duflo 
posing with or among black African and Indian children from poor communities. 
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research continues to appear, suggesting that there may be something 
akin to ethical incommensurability: there are groups of people, including 
scholars, who are of the view that certain experiments are inherently 
unethical and reflect patronising and harmful views of the experimental 
subjects, whereas the scholars and funders behind these experiments see 
them as benevolent, subtle and with positive repercussions. 


A very recent example (Barsabai et al. 2022) concerns an experiment 
involving domestic workers from the Philippines working in Hong Kong 
and Saudia Arabia, who are recognised as being exploited. The researchers 
got these workers to give their new employers a packet of dried mangoes 
and photographs of their (the workers’) families in order to reduce the 
‘perceived social distance’ between workers and employers, thereby 
improving treatment of the former by the latter. Evidently, this must have 
seemed like not only a good idea to the researchers, ethical review board 
and funders for it to get to the stage of implementation but to be such a 
good idea that it merited the research resources devoted to it. Yet as 
soon as the working paper was published, other scholars and individuals 
expressed outrage on social media at what they took to be an obviously 
flawed and harmful experiment. A common theme was along the lines of 
‘how did this get conceived/approved/published?’. The extent of the 
difference between these two groups appears to reflect completely 
different worldviews, and it is notable that critics tend to be scholars from 
outside economics whose disciplines more explicitly address issues of 
epistemic asymmetries and harms, along with people from developing 
countries. Even those who take the view that such experiments are not 
unethical or harmful are nevertheless obliged to accept that this difference 
exists and must necessarily manifest in the research questions chosen - 
linking to the earlier concern about smuggling in normative biases through 
notionally objective and ‘scientific’ research. Yet they have no intellectual 
basis for dealing with such differences other than to simply dismiss 
opposing views.” On the other hand, those who take the view that local 
knowledge acquired by other means may be more accurate or insightful 
than experiments by outsiders typically also work within (formal or 
informal) intellectual frameworks in which both the existence and the 
wrongness of the randomista approach are explicable. The strikingly 


27. This is a key difference between this example and the much-discussed case of Kuhn’s ([1962] 1996) 
assertion that scientific paradigms were incommensurable. That statement was more a problem for realists 
than anti-realists since it could be accommodated by the latter’s intellectual framework. However, in that 
debate the former focused on denying that in fact scientific paradigms are incommensurable - something 
that their intellectual framework provided resources for. Economists do not have the intellectual resources 
to construct an argument that the two sets of views on the (un)ethical nature of certain experiments are 
reconcilable in a way that supports their stance. 
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different views on ethics serve as further substantiation for critics, 
whereas for proponents they are inexplicable.”® 


E Addressing potential objections 


A common objection to certain of the criticisms and concerns raised earlier 
is that they are now, as of 2023, ‘out of date’. In other words, that these 
have been acknowledged and addressed by the majority of advocates of 
RCTs, especially the more prominent ones. This is false: the validity of the 
criticisms and concerns has typically not been acknowledged, and where it 
has the nature of the acknowledgement has not done justice to the 
substantive implications for recent and future advocacy of RCTs. 


For example, it is quite common to claim that substantial progress has 
been made recently on the problem of external validity (generalisability). 
Such claims typically cite recent work by the likes of Meager (2019, 2022) 
and Vivalt (2015, 2020), which attempts to apply the logic of meta-analysis, 
long-used in other disciplines, to such problems within an econometric 
framework. However, no consensus exists on how meta-analysis should be 
used to address the external validity challenge and earlier work already 
argued that replication, which lies at the heart of the meta-analytic 
approach, cannot resolve the fundamental and fatal problem posed by 
external validity for the randomista case (Muller 2015, 2020). 


Furthermore, there has been no acknowledgement of what the claim of 
substantial progress would imply for all preceding uses of RCTs to inform 
policy, including those awarded the 2019 Nobel Memorial Prize. The 
criticisms that supposedly have been addressed imply that past policy 
recommendations were almost entirely inappropriate and probably wrong 
because they failed to deal with external validity in any serious way. Yet, 
there has been no such acknowledgement by practitioners or proponents. 


Another objection is that policy-makers are not as much in thrall of RCTs 
as critics suggest: in effect defending the ‘credibility revolution’ by claiming 
there has not really been a ‘revolution’. While defenders of the randomista 
approach tend to merely assert this claim, there are an increasing number 
of thorough assessments that find otherwise. Some are by critics 
(Bédécarrats et al. 2019), but others are by those very sympathetic to the 
randomista case. For example, Mehmood, Naseer and Chen (2021) find 
that, in Pakistan, ‘Cecono)metrics training shifted policy-makers’ beliefs 
towards the paradigm associated with the credibility revolution’. A notable 
aspect of the randomista approach through organisations like J-PAL is 


28. By this, | mean substantively inexplicable: economists often explain criticism of their approaches by 
explicitly or implicitly suggesting their critics are simply ignorant. 
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their concerted effort to train researchers and policy-makers in developing 
countries in its arguments and methods. 


Other objections fail similarly because they are, in effect, superficial. 
For instance, some argue that the claim of RCTs being a ‘gold standard’ 
is now out of use. This is true, but the mere discarding of the term is not 
the material issue. The notion that RCTs are at the top of an evidence 
hierarchy remains deeply entrenched in assessments of the relative 
‘rigour’ and ‘credibility’ of different kinds of evaluations, which amounts 
to the same thing. 


The concern about the questions addressed by RCTs being effectively 
determined by methodological preference rather than societal important 
or broader understanding of social problems remains unaddressed. Added 
to that is the further unacknowledged problem of how the choice of 
intervention itself smuggles in subjectivity and bias into a process that 
purports to be ‘objective’ and ‘scientific’ (Muller 2022), 


A final common objection is that the concerns raised apply to other 
approaches as well, not just RCTs. For example, other formal (econometric 
and statistical) approaches have tended to devalue qualitative methods 
and local knowledge. They have also not adequately addressed the external 
validity problem. This is certainly true in many cases, but it in no way refutes 
concerns that the use of RCTs, and the randomista approach in particular, 
does not deliver what it claims to and is likely to do harm to the goal of 
creating better societies rather than enable progress towards it. 


In short, all the concerns raised in earlier sections remain relevant and 
have not been adequately addressed or acknowledged - despite rhetorical 
efforts by proponents of these methods and approaches to suggest 
otherwise. 


E Conclusion 


The broad set of concerns - of subjectivity, implicit marginalisation of other 
forms of knowledge and entrenching or expanding North-South 
asymmetries - are increasingly being raised as the prevalence and influence 
of RCTs becomes more apparent. However, there has been relatively little 
detailed examination or investigation of the various components of these 
concerns. 


The AfrED database and others like it can be used to identify patterns in 
the kinds of interventions that are evaluated and, correspondingly, those 
that are omitted. This can shed light on at least two different issues. The 
first relates to methodological exclusions, in which certain kinds of research 
questions are omitted because they cannot be addressed by a favoured 


213 


Why emphasis on randomised controlled trials is a flawed approach to evaluation 


method. (The ‘searching under the streetlight’ problem). The second is 
whether the kinds of interventions considered in relation to a given research 
question are of a particular type or set of types and, in turn, whether those 
correspond to prior beliefs about socio-economic relations, mechanisms 
and parameters. The omission and neglect of such issues to date are 
advantageous for ex-post analysis of this kind, as it means that there will 
have been little effort to conceal such leanings or attempt to give the 
impression of balance.?° 


Given information on funding sources the database already provides 
information that corroborates concerns about North-South asymmetries. 
As the database website notes, ‘These reports are typically commissioned 
and funded by international (development) agencies [...] or foreign 
governments [...] A smaller number of reports are commissioned by African 
governments’ (AfrED 2022). Such observations could be elaborated upon 
in more detail and linked to some other contributions on the asymmetries 
in the locations of researchers involved in such initiatives and scholarly 
work (Chelwa 2021; Panin 2020). What has not been examined is whether 
other characteristics of evaluations correlate with institutions involved. Are 
there differences between the kinds of programmes implemented 
(evaluated) by African country governments themselves relative to donors? 
Is there variation in the approaches taken to evaluating those programmes? 


Moreover, there are important nuances that must be attended to by 
such analysis as there has been a concerted effort by institutions such as 
J-PAL, 3ie, DIME and others to secure the direct support and co-operation 
of developing country governments for their approach. Over time, then, we 
would expect that the success of such efforts would narrow the gap 
between the kinds of evaluations commissioned by the different types of 
institutions and governments or government agencies. And, indeed, it may 
be that this reproduces earlier dynamics in the evaluation space so that, in 
some respects, RCTs emerge as the most recent manifestation of these 
structural patterns and processes. 


Using the database to examine ethical issues and disagreements 
would likely be less straightforward in as much as it would require 
additional information. For instance, one could select a subset of 
evaluations and ask different respondents, in formal or more accessible 
parlance, what they think of the ethics of those interventions (and 
possibly how they are evaluated). Combining those responses with a set 
of questions about respondent characteristics could provide further 
interesting information on correlates of differences. Alternatively, one 


29. In the terms economists like to use: the patterns of these choices should (in a qualitative way though it 
may be possible to quantify some aspects) reveal the preferences of those making the decisions. 
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could examine characteristics of the interventions themselves as 
correlates for these differences. For example, whether interventions or 
evaluations commissioned or conceived by particular actors are more 
likely to be deemed unethical or otherwise problematic by ‘ordinary 
citizens’ in the relevant countries. 


Despite the raft of damning critiques of the randomista approach, 
many critics have noted that there is no sign of its influence waning, 
consistent with the view that this influence was never derived primarily 
from the approach’s substantive merits but rather rhetoric, resources, 
power and misplaced epistemic authority. Despite that, it bears asking 
what broader lessons might be drawn for the field of evaluation from the 
failings and harms of the randomista approach. In brief, | would suggest 
the following in order for evaluators to mitigate their complicity in harm 
and better achieve the objective of building just and equitable societies. 
Firstly, even where a given evaluation approach is indeed the best possible, 
it may nevertheless not be able to definitively answer the questions we 
have because of the nature of societal dynamics and structure.*° Secondly, 
that historical experience repeatedly reveals claims that a given 
quantitative research method dramatically and comprehensively resolves 
the difficult questions of societal understanding and policy to be 
exaggerated at best and harmfully flawed at worst. Thirdly, the choice of 
intervention or evaluation is informed by prior beliefs about societal 
phenomena and how they might be influenced; the notion of value- 
freeness is simply a myth in such contexts (Douglas 2014), and the 
philosophical literature that mainstream economists sometimes gesture 
at is more than half-a-century out of date (and even then is incorrectly 
interpreted). Fourthly, evaluation takes place within a broader 
epistemological context that is influenced by power and resources, often 
reproducing dynamics analogous to those of colonial and other 
undemocratic or oppressive contexts. While those conducting individual 
evaluations may be unable to influence these broader dynamics, or even 
speak to them given the possible consequences for funding and 
subsequent work, they can nevertheless reduce their own role in causing 
harm by adjusting the strength and tone of conclusions and 
recommendations. An evaluator who makes a definitive claim about a 
phenomenon based ona single RCT while ignoring other kinds of evidence 
is complicit in epistemic harms, but it is quite possible to produce clear 
conclusions regarding what a single RCT has found without then making 
assertions of that overly broad kind. The same is true for other approaches 


30. For more on this see Muller (2021b) on the ‘ontological horizon’ versus ‘epistemological horizon’ across 
disciplines. 
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to evaluation. The case for such ‘epistemic humility’ (Angner 2020; Manski 
2011) extends to a wide range of contexts. 


There is more than enough scholarly work which credibly and 
convincingly demonstrates that the randomista approach is flawed on its 
own terms (Muller 2020) and within broader epistemological frameworks. 
These flaws limit and distort the understanding of inequality and inequality- 
generating mechanisms while also generating harms and unwarranted 
inequalities in the process of influencing and determining policy. The 
appropriate response would be for that approach, and RCTs in general, to 
be treated as a research method with many weaknesses and some strengths 
and not at the top of any evidence or epistemic hierarchy. Evaluators need 
to be aware that the RCT-based approach as it is currently deployed is 
seriously flawed, but, moreover, even if implemented more responsibly, is 
still very limited in its capacity for enabling our understanding of inequality. 
Unfortunately, it appears that greater effort will be needed to achieve a 
change in attitude towards the approach that goes beyond the actual 
merits and challenges the rhetoric and deliberately constructed institutional 
power of RCT proponents. Proponents have back-pedalled substantially in 
the last decade and weakened a range of claims but have done so in a 
manner that seeks to obscure these changes, thereby muddying the waters 
for most observers (Muller 2021a). Nevertheless, it seems inevitable that 
eventually, the ‘credibility revolution’ narrative will fade into obscurity to be 
replaced by the next overhyped approach.” The critique presented earlier 
is consistent with broader concerns about the dominance of the ‘white 
gaze’ in the evaluation of international development initiatives (Shallwani & 
Dossa 2023), the decolonisation of which ‘requires a critical reflexive 
analysis of the assumptions, motivations and values that underpin evaluation 
practices’. What is most important, then, is that the lessons from the flaws 
and failures of the randomistas be reflected in, and incorporated into, the 
evaluation literature. One problem is that ‘intellectual dependency’ means 
that African scholars and policy-makers tend to follow trends in the Global 
North with a lag of a decade or two, which means it could be many decades 
before this happens. Given that many of these problems impact African 
countries and peoples more than many others, there is no reason why 
African scholars, researchers and evaluators should wait for the literature in 
the Global North to finally shift: it is well past time for more epistemological 
critique to be generated and disseminated from the Global South on 
matters that affect its peoples so profoundly. 


31. An example within economics, that may or may not make the jump to the evaluation literature, is 
machine learning (Athey & Imbens 2019; Muller 2021a). 
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E Introduction 


National and international organisations and initiatives must address equity 
and sustainability as core criteria for evaluation at every level from the local to 
the global, for all projects and programmes. Making equity and sustainability 
universal evaluation criteria means taking them seriously, tracking them over 
time, making comparisons, generating findings and drawing lessons to inform 
future initiatives. This is how evaluation contributes to a more just, sustainable 
world. (Patton 2021, p. 32) 


The South African Constitution provides for a unified and equitable society 
in which human dignity and rights are upheld for all its citizens.** The 
National Development Plan 2030 (NDP) further clarifies a vision of South 
Africa that is ‘just, fair, prosperous and equitable’ by 2030 (NDA 2012, 
p. 61). However, nearly 30 years after the transition from the apartheid 
system to a democratic government, South Africa continues to align this 
vision with its lived reality. 


The history of colonialism and apartheid has left deep-rooted injustices 
and disparities in access to and ownership of services, resources and assets 
(World Bank 2022, p. 3; Makgetla 2020, p. 4), while the post-apartheid 
economy continues to reproduce extreme levels of inequality in income 
and wealth (Francis & Webster 2019, p. 789). The coronovirus disease 2019 
(COVID-19) pandemic exposed the depth and breadth of the country’s 
structural inequities and strengthened the mandate to redress the causes 
of inequity for a more just and equal society (Francis, Valodia & Webster 2020). 
While the government has responded at scale, with over 18 million people 
on social grants and 7.5 million on the temporary COVID-19 Social Relief of 
Distress Grant, issues of inequity and related urgent issues like climate 
change and ecosystems breakdown are becoming increasingly pronounced 
across South Africa, as evidenced in the unrest in the KwaZulu-Natal (KZN) 
and Gauteng Provinces in July 2021, the taxi strike in Western Cape and the 
extensive damage caused by the flooding in April and May 2022 in the KZN 
and the Eastern Cape provinces and June 2023 in Western Cape. Neglecting 
to transform the existing systems and structures that replicate inequities 
threatens the full development and stability of the country (Hickel 2022; 
Vhumbunu 2021). Truly resolving these systemic issues requires ‘whole of 


32. Constitution of the Republic of South Africa, 1996: Preamble, s. 3 (2a), s. 7 (D, s. 9, s. 10 
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society’ action, including the mainstreaming of issues of equity and climate 
and ecosystems health (CEH) into all aspects of policies and programmes, 
including their evaluation. 


Evaluations have transformative potential because, as American 
evaluation theorist Ernest House argues, evaluation can be used to either 
shift or maintain existing repressive structures, as findings are ‘used to 
determine “who gets what” and that [...] evaluation’s primary purpose is 
[...] to promote social justice’ (Christie & Alkin 2013, p. 38). Further, 
evaluation plays a critical role in transformation by assessing how and to 
what extent interventions address systemic inequity and promote equitable 
development (Bitar 2021, pp. 1-2; Chaplowe & Hejnowicz 2021, p. 2). 


Evaluators (and the evaluation process itself) can facilitate systemic 
change for greater equity and ecosystems regeneration through the way 
the evaluation is designed, the evaluation questions raised, the approaches 
used for data-collection and validation, stakeholder engagement processes, 
the recommendations emanating from the evaluation results and how 
results are disseminated and advocated for evidence use. Evaluations that 
focus on single-loop learning (e.g. was the intervention implemented as 
planned, i.e. done right?) are unlikely to be transformative because they 
lack depth in analysis. Evaluations that seek double-loop learning (e.g. 
were the right things done?) that question values, or triple-loop learning 
(was the underlying paradigm right?), which questions the why or the who 
(Aston 2020; Tamarack Institute n.d.), have much more transformative 
potential. 


In South Africa, the Department of Planning, Monitoring and Evaluation 
(DPME)’s National Evaluation Policy Framework (NEPF) 2019-2024 
presents the government’s commitment to evaluation as a mechanism for 
promoting equity and achieving the NDP 2030. The NEPF outlines the 
need for government interventions to be equity-responsive (DPME 2019, 
p. 13) and stipulates principles that underpin the selection of evaluations, 
including the principle of inclusivity. Yet, a recent review of the National 
Evaluation System (NES) highlights that many evaluations struggled to 
adequately incorporate an equity focus and did not sufficiently address 
issues affecting females, youth, persons with disabilities and other 
marginalised groups (Goldman et al. 2019). To address this gap, DPME 
developed the Gender-Responsive Planning, Budgeting, Monitoring, 
Evaluation and Auditing Framework (DPME 2019). However, it focuses 
primarily on gender and disability and does not account for how wider 
systemic factors create and perpetuate inequity, nor does it examine how 
inequities exacerbate the negative consequences of climate and ecosystems 
breakdown (Hickel 2021; Raworth 2017; Schmelzer, Vetter & Vansintjan 
2022). The confluence of these two issues is recognised now in the 
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South African state’s commitment to a Just Transition, as represented in 
the recently approved Just Transition Framework.*? 


During arecent virtual evaluation hackathon hosted by the South African 
Monitoring and Evaluation Association (SAMEA), two hackathon teams 
developed two new evaluation criteria and accompanying guidelines, 
which will be issued by DPME for broad application across South Africa, to 
address the systemic crises of climate and ecosystems breakdown and 
systemic inequity. The groups’ hypothesis was that if evaluation criteria 
could be developed to focus on these key issues, with guidelines ratified by 
DPME, and if the criteria are applied in all evaluations, then issues of equity 
and CEH would be mainstreamed into the evaluation system, thereby 
contributing to the transformation of policies and programmes to address 
these systemic crises. 


This chapter focuses on the development of the Transformative Equity 
evaluation criterion and guideline, providing a discussion of the criterion 
and how it was formulated. As this criterion was developed alongside the 
criterion on CEH, some references to this other criterion will be made. The 
chapter also discusses the pilot process underway, through which we are 
following government agencies’ attempts to integrate the criteria into their 
evaluation terms of reference (TORs) and evaluation implementation 
process, so testing our hypothesis. Finally, there is a reflection on initial 
lessons learned from the pilot and how the criteria and guidelines are 
influencing departments’ mindset regarding how to more extensively 
transform our monitoring and evaluation (M&E) approaches to promote a 
more just and sustainable society. 


E Understanding inequity in South Africa 


South Africa is the most unequal country in the world%* (World Bank 
2022) and the levels of inequality have not significantly changed since the 
end of apartheid (Francis & Webster & 2019, p. 788). In the South African 
context, economic inequality is extreme, with 10% of the population 
controlling over 85% of the wealth (Chatterje et al. 2020: in Leibbrandt & 
Pabon 2021, p. 179). The history of apartheid contributed to high levels of 
inequity aligned with racial groupings. Statistical data from 2011-2015 
demonstrates that the average real earnings for black Africans was less 


33. https://www.climatecommission.org.za/just-transition-framework 


34. Based on World Bank assessments of the Gini coefficient, which is a measure of income distribution 
across segments of society and is used to demonstrate levels of inequality, South Africa has the highest 
level of inequality, at 0.69 out of a scale of O-1 out of 149 countries for which there is data. 
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than ZAR7,000 per month, while other racial groups earned between 
ZAR2,000 to ZARI17,000 per month (Maluleke 2019, pp. 61-62).%° 


Inequity in South Africa also manifests along other social factors, 
including gender, age and ability. The pre-1994 administrations presided 
over a state that supported a separate development (apartheid) system 
that prioritised economic zones and traditionally white European economic 
centres at the expense of indigenous populations (Prinsloo 2021). This 
decision provided the legal basis for widespread injustices in South Africa 
and exacerbated the spatial, racial and economic inequalities, poverty and 
unemployment, particularly in the so-called bantustans or ‘homelands’ 
(Prinsloo 2021). Between 1994 and the mid-2000s, in the early years of 
democracy, South Africa was able to achieve its best years of economic 
growth because of various macroeconomic policies, like the Reconstruction 
and Development Plan (RDP), which were designed to achieve a 
transformation in South Africa by reducing the ‘triple challenges’ of poverty, 
inequality and unemployment (DPME 2014, p. 83). 


The legacy of apartheid continues to contribute to the inequality in 
South Africa, which is manifested in the highly disparate quality of 
infrastructure and services between different geographic areas, particularly 
the former homelands and townships. The policies and systems that were 
established during apartheid continue to affect the availability and 
accessibility of quality education, infrastructure, basic services, employment 
and markets across different segments of society, further perpetuating 
social and income disparities (Makgetla 2020; Shifa, David & Leibbrandt 
2021). The spatial and economic inequalities also intersect with 
environmental vulnerabilities, such that households living in labour-sending 
areas (Makgetla 2020, p. 28), former homelands and informal settlements 
are much more likely to be affected by environmental and climatic events 
or the risks of pollution and other environmental degradation than 
households in more affluent areas (Venter et al. 2020; Williams et al. 2019, 
p. 158). Moreover, various redistributive policies, like the broad range of 
social grants and Broad-based Black Economic Empowerment (BBBEE), 
have yet to reduce inequities in the country. 


The NDP 2030 acknowledges key development challenges that require 
focused attention in order to achieve the state’s transformational 
development objectives, including inadequate infrastructure, spatial 
divides and persistent poverty. These development challenges reflect 
paradigms for understanding how inequality and inequities manifest and 


35. Although race is major factor in the high levels of economic inequality in South Africa, it has been noted 
that inequality in South Africa is also becoming more based on class (Webster, Valodia & Francis 2017). 
Makgetla (2020) points to data showing that in the richest 5% of households, 45% were white, 42% black 
African, and 13% coloured or Asian. 
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are reproduced (see Makgetla 2020) for more on how inequalities are 
reproduced). Supporting the NDP, the NEPF 2019-2024 centres the 
importance of evaluation in policy-making and management and links 
evaluation to planning and budgeting processes, with specific emphasis on 
improving performance on government objectives for greater equity, 
inclusion and social development. 


E Development of the evaluation criterion on 
Transformative Equity 


With the inability to hold a physical conference in 2021 because of the 
ongoing COVID-19 pandemic, SAMEA facilitated a two-week virtual 
hackathon* to co-create solutions for key issues facing M&E and its 
contributions to society. The main themes were identified and included the 
following: (1) addressing systemic crises, including inequity and climate 
and environmental degradation; (2) how to implement M&E during a crisis 
(including designing rapid evaluations and conducting evaluations 
virtually); (3) MAE and producing an evaluation evidence map for South 
Africa. Two teams engaged on the theme of systemic crises, one focusing 
on issues of equity and one focusing on the need for environmental 
regeneration. Given the importance of evaluation criteria to evaluations, 
each team was tasked with developing a criterion and supplementary 
guidelines for commissioners and evaluation practitioners on how the 
criterion could be integrated into evaluation. 


The team tackling the criterion and guideline on equity comprised 
eighteen people from SAMEA, the DPME, the National Department of 
Social Development (DSD), the National Development Agency, independent 
evaluators, researchers and civil society representatives. The team met 
virtually for two hours a day over the two-week period. They reviewed 
existing literature on the theoretical underpinnings for evaluation as a 
mechanism to support transformation and equity, interacted with key 
experts on the context of high levels of inequities in South Africa, and 
debated and defined the criterion, structure and content of the guideline 
before developing the core content for the guideline document. Key 
readings included recent work by Chaplowe and Hejnowicz (2021), Michael 
Quinn Patton (2020), Khalil Bitar (2021), the collection of contributions in 
Transformational Evaluation ( Van der Berg, Magro & Adrien 2021), economic 
and social analysis by Trade and Industrial Policy Strategies (TIPS; Makgetla 


36. Key champions to this undertaking included South Africa’s Department of Planning, Monitoring and 
Evaluation (DPME) and JET Education Services, with funding support from UNICEF South Africa and the 
National Association of Social Change Entities in Education (NASCEE). 
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2020) and the Organization for Economic Co-operation and Development 
- Development Assistance Committee (OECD-DAC) criteria. 


Michael Quinn Patton argues that ‘criteria constitute the nuclear core of 
evaluation’s energy function: rendering judgement. Without criteria, there 
can be no judgement. Without judgement, there can be no evaluation [...]’ 
(Patton 2020, p. 4). Criteria enable evaluators to prioritise what is important, 
identify the key questions to ask, the types of data to collect and how to 
interpret results (ibid). The most widely used evaluation criteria are those 
set by the OECD-DAC Network on Development Evaluation (EvalNet): 
relevance, coherence, effectiveness, efficiency, impact and sustainability 
(OECD 2019). However, numerous evaluation experts have questioned the 
DAC rationale that issues of equity and inclusion can be covered in other 
domains, such as relevance, effectiveness and impact, rather than 
developing a unique criterion on equity during its most recent revision 
phase (Bitar 2021; Chaplowe & Hejnowicz 2021; Ofir 2021; Patton 2020). As 
Ofir (2021) reflects, ‘the criteria are fine for “business as usual summative 
and accountability evaluations,” but remain inadequate for addressing 
systems change and transformation from local to global level’. Given the 
importance of criteria in guiding the evaluation process, the hackathon 
team decided that an explicit criterion was indeed needed to promote 
greater equity in and through evaluations. 


A key decision in determining the name of the criterion was to use the 
concept of equity over equality and that achieving equity requires 
transformation. The hackathon team spent several sessions exploring 
personal assumptions and understandings of equity and equality in the 
context of South Africa. The team recognised that equality and equity are 
often conflated yet distinct concepts. Equality can be understood as a 
state of affairs in which all individuals, groups or areas receive the same set 
of benefits or have the same exposure to opportunity regardless of their 
current position of privilege or need (Espinoza 2007, p. 345; Minow 2021, 
p. 174). Equity refers more to fairness and justice within social and economic 
systems (Bamberger & Segone 2011, p. 3), and the extent to which persons 
or regions receive appropriate levels of support according to their level of 
need (Minow 2021, p. 173). Equity also recognises that systems have been 
purposefully designed to benefit some individuals, groups or areas over 
others and that these other entities require differentiated interventions so 
that they can equally benefit or participate (Espinoza 2007; Minow 2021, 
p. 174). Achieving equity often requires a transformation of systems to 
break apart the structures that perpetuate the imbalances (Mertens 2021, 
p. 3; Minow 2021) and bring them to a state of fairness. Given the high 
levels of inequality in South Africa, the team recognised that the emphasis 
of the criterion needed to be on promoting equity and justice in order for 
the country to reach real equality. 
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Following the discussion on equity versus equality, the hackathon team 
determined to name the criterion ‘Transformative Equity’. While the 
criterion and dimensions will be described in more depth in the next 
section, it is important to note that the team decided to define the criterion 
using five dimensions (e.g. who, how, where, what and when) to reflect the 
specific context of South Africa and to help evaluation commissioners, 
managers and practitioners consider how issues of equity play out broadly 
within intervention settings and in evaluation processes. 


In addition to defining the evaluation criterion, the hackathon team 
started developing a guideline to support the application of the criterion in 
evaluations. In discussing how the equity criterion would be defined and 
how the guideline should be written to assist application, the team agreed 
that for the criterion to promote transformation, the guideline must 
motivate evaluation commissioners, managers, practitioners and users to 
adopt an ethical stance that prioritises inclusivity, respect for human rights 
and dignity, advocate for social, economic and environmental justice and 
be willing to challenge the existing power structures (see Mertens 2009, 
2021). As the intention was for the DPME to issue the guideline as the owner 
of the NES, the team decided to use the DPME’s guideline on developing 
TORs (DPME, 2016) as the structure for the guideline, providing examples 
of how the equity criterion could be integrated into the design, 
commissioning and implementation of an evaluation. The guideline 
demonstrates how evaluation purpose statements and evaluation questions 
could be rephrased to include a focus on equity, how the methodology 
could be expanded to include the marginalised and disempowered, and 
restated the importance of developing and using evaluation findings that 
promote transformative change. 


Following the hackathon, a smaller team, including the authors of this 
chapter, worked on finalising the guidelines. The first draft was shared with 
fifteen external peer reviewers in April 2022, including representatives from 
DPME, DSD, United Nations Children’s Fund (UNICEF) and independent 
evaluators. Feedback was incorporated into the final version of the 
guideline, published on SAMEA’s website in July 2022.7 A pilot process 
was then initiated to test the practical application of the guideline. This 
entailed the identification of upcoming evaluations where the TORs of 
these could be adapted to include the new criterion, with the staff 
responsible for these evaluations participating in a two-day training session 
in July 2022. Following the training, it was expected that the TORs would 
be adapted and evaluations commissioned. The pilot process is further 
discussed later in this chapter. We now describe the criterion. 


37. Currently available at https://www.samea.org.za/evaluation-hackathon-reports. 
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Transformative Equity criterion 


The Transformative Equity criterion is defined as ‘the extent to which an 
intervention’s objectives, design, implementation and impact contribute to, 
or do not contribute to, addressing systemic inequities and promotion of a 
more inclusive society’. By deliberately including the word ‘transformative’, 
the criterion underscores the assumption that reaching equity requires a 
transformation of systems and structures, both in the intervention and in 
the evaluation. Furthermore, the emphasis on systemic inequities draws 
attention to the way systems have been designed, established and 
maintained to perpetuate inequities. 


To help users better understand the criterion and how it could be 
applied, the team identified five dimensions through which equity and 
inequities manifest: Population/oopulace, Cause and effect, Space, Content 
and intention, and Timing. These five dimensions were proposed based on 
the need to understand the intervention’s context in relation to the history 
of South Africa, the readings on factors reproducing inequity and the need 
to promote transformation. These are described later on. Later sections will 
describe how these dimensions can be applied in evaluation. 


O Dimension 1: Population/populace: Who? 


The ‘Who’ dimension reflects on which segments of the population are 
prioritised or excluded by an intervention to examine who benefits and 
who loses (whether intentional or not) and how the consequences of an 
intervention - expected or unexpected - affect groups differently. Critical 
in the context of Transformative Equity is the intentional inclusion of 
individuals or groups who may be disempowered or disenfranchised, the 
historically marginalised and otherwise voiceless. In addition to the groups 
commonly identified in discourse about inequality (e.g. females or racial 
groups), ‘who’ encourages the consideration of persons with disabilities, 
migrants or refugees, and those of diverse sexual orientations while also 
requiring an intentional awareness of how power and inclusion unfold in 
the intervention (and the evaluation) and on how power results systemically 
in negative consequences for different groups in society. Attention to the 
intersections between different categories is critical, for example, rural 
African females or young African trans-persons, when considering who is 
included or excluded and who benefits or loses. 


38. Ability and skill to mainstream the equity criterion in the evaluation process is also critical to ensure an 
equity lens is applied to evaluation. The skills and qualities needed of evaluation technical working groups 
and evaluation teams is described later in the chapter. 
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O Dimension 2: Cause and effect: How? 


The ‘how’ dimension requires a critical consideration of three aspects. 
Firstly, the context within which an intervention operates, particularly the 
systemic issues that perpetuate inequities. Secondly, the extent to which 
the design or implementation of the intervention is responding to inequity. 
Thirdly, the dimension requires a consideration of the long-term effects of 
the intervention on equity. In this way, the ‘how’ dimension reflects an 
element of relevance or appropriateness to the broader context within 
which the intervention is positioned. 


‘How’ requires consideration of structural drivers that perpetuate social 
and economic inequities and the extent to which the intervention adequately 
and appropriately addresses these factors. Examples of such structural 
drivers include inequalities in the ownership, access to and control of 
assets, large disparities in wages and disparities in the quality of education 
between schools in former homelands and all-white schools. These 
structural drivers are remnants of the apartheid system that continues to 
influence access to and the distribution of wealth, jobs and services. 


In understanding how interventions are responsive to issues of inequity, 
this dimension suggests that evaluations should focus on examining the 
extent to which the intervention was delivered in a manner that was aligned 
with the needs and expectations of the different priority groups, the 
flexibility of the intervention to provide differentiated components 
depending on location or group need and the extent to which it seeks to 
change the structural issues which disempower. 


O Dimension 3: Space: Where? 


The ‘where’ dimension focuses on spatial and geographic contexts of the 
intervention (e.g. space, quality of the environment and rights of access or 
use by different groups), the way in which the intervention is experienced 
in different geographic areas and the extent to which the intervention is 
able to address the causes of inequality that result from these spatial or 
geographic contexts. Given the history of apartheid, this dimension 
recognises the persisting spatial inequities and their intersection with other 
manifestations of inequity. The differential effects of interventions play out 
in different communities such as rural, informal settlements, formalised 
peri-urban townships and traditional formal urban centres. Of particular 
concern are the former homelands, which are some of the poorest areas in 
South Africa. 


‘Where’ recognises that different locations or areas provide greater or 
lesser opportunities for people in terms of education, economic opportunity, 
mobility, health and environment and neighbourhood quality because of 
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multiple reasons, including historic factors, geographic location and 
available environmental resources. It also assesses how the intervention 
seeks toaddress the geographic and environmental issues that disadvantage 
particular places and social groups, for example, designing appropriate 
services for rural areas, thus speaking to issues of environmental justice. 
Equity can also be an important consideration in the sustainability of 
results, for example, the degree to which the benefits of a programme or 
intervention are sustained may also depend on the socio-economic or 
spatial or geographic traits of the focus community. 


O Dimension 4: Content and intention: What? 


While some interventions have explicit objectives to promote equity 
or inclusion - for example, a youth employment programme - the 
Transformative Equity criterion urges that every evaluation includes an 
assessment of an ‘intervention’ objectives and results to determine if and 
how they contribute to equity and social development. Therefore, the 
‘what’ dimension urges evaluators and other key stakeholders to consider 
the choice of intervention and whether the design and implementation 
approach adequately aligns with transformative objectives. 


Specifically, ‘what’ urges evaluators and other key stakeholders to 
consider the extent to which the intervention is meeting or contributing to 
(or expected to contribute to) specific transformational objectives that 
seek to redress social and economic inequities; to differentiate between 
interventions that make a deliberate and concerted effort to address issues 
of equity versus those that lightly glance over it and whether interventions 
seek only to address symptoms but not the root cause of the problem. The 
‘what’ dimension requires reviewing or developing the intervention’s theory 
of change in order to determine the driving objective of the intervention 
and how equity considerations are integrated into the theory of change 
and theory of action of the intervention. In considering issues of effectiveness 
of interventions in achieving equity objectives, it is important for evaluators 
to distinguish between theory failure and implementation failure, 
recognising that the evaluation should consider both how well the 
intervention was designed and delivered in terms of addressing equity 
issues. 


O Dimension 5: Time: When? 


The ‘when’ dimension urges evaluators and commissioners to consider 
various time elements with regard to the intervention and the evaluation 
process. Firstly, the ‘when’ dimension urges evaluators and commissioners 
to understand the period when the intervention is taking place, paying 
attention to the incentives in the period within which the intervention and 
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the evaluation are situated. For example, there may be more openness for 
certain social changes after an election than would be present just before 
an election. Similarly, given the attention placed on high levels of youth 
unemployment, how would an evaluation of the Youth Unemployment 
Service (YES) programme be influenced by timing? In this way, the ‘when’ 
dimension relates to relevance and coherence. 


In addition, ‘when’ encourages evaluators and commissioners to consider 
how the equity issue under consideration has changed over time. This 
consideration involves a retrospective scan to deepen the understanding 
of the duration of the issue or periods when the issue has been more intense 
than others. Evaluators can be asking, ‘Has there been any improvement in 
the issue over time? If not, why not?’ For example, an evaluation of a land 
reform initiative would include a review of previous policies, programmes 
and evaluations in the contextual analysis. 


Finally, evaluators and commissioners are encouraged to think about 
the durability of equity results over time. If interventions are to be truly 
transformative, the changes in equity need to last into the foreseeable 
future. 


Equity principles 


In considering how the criterion would be applied in practice, the hackathon 
team recognised that a set of principles may be needed to assist 
commissioners, evaluation managers and evaluators as they conduct their 
work. These principles were derived from the existing literature on human 
rights and equity, power and participation, systems thinking and the 
relational African philosophy of ubuntu. They also require that the evaluator 
is not neutral but is actively seeking to address problems of systemic 
inequality through the evaluation process. In the drafting process, the 
hackathon team identified five key principles and then described how these 
principles could be applied as a commissioner or manager of evaluation 
and as the evaluator: 


1. Equality, justice and respect for human dignity: The South African 
Constitution upholds the equal enjoyment of rights and freedoms for 
every person in South Africa and that no person should be disadvantaged 
or discriminated against. This understanding of equality is based on 
the belief that all persons have the right to have their human dignity 
recognised, protected and respected. In the realm of evaluations, 
evaluators, commissioners and funders need to pay attention to the ways 
in which the intervention and the evaluation process itself promotes the 
principles of equality and respect for human dignity. They are urged 
to uphold impartiality, seek and promote justice, treat all stakeholders 
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with the same level of respect and ensure that less powerful stakeholder 
groups have equal opportunities to make their voices heard or to review 
and respond to evaluation findings and recommendations. 

2. Awareness of power and voice: Power dynamics are hugely influential 
both in the implementation of interventions and in the evaluation of 
such interventions (Rodrigues-Bilella, Salinas Mulder & Zaveri 2021; 
Sibanda & Ofir 2021). Power differences exist between implementers 
and participants, between evaluator and participants, between 
commissioners and evaluators and between funder and recipient. To 
integrate equity into an evaluation, intentional awareness of, and 
deliberate enforcement of, and ability to mitigate against, dynamics that 
undermine fairness and equality is required. Evaluators, commissioners 
and funders must be prepared to take on the challenge of explicitly 
unpacking and acknowledging how power manifests and need to be 
willing to change their perspective from ‘power-over’ to ‘power-with’ 
(Abidi-Habib et al. 2021, p. 109; Pansardi & Bindi 2021, p. 51). The 
continuous and intentional awareness of power and voice is expected to 
facilitate a inclusive and equal share of the evaluation process by 
evaluators and commissioners to ensure ownership of the evaluation 
process such that study participants and others invested in the 
intervention have the opportunity to contribute to the evaluation 
planning, design, methodology, interpretation of findings and 
identification of recommendations as well as the use of the evaluation. 

3. Ubuntu: Ubuntu emphasises the relational way of being that 
acknowledges interdependencies between persons, and between all 
things, living and non-living, material and spiritual (Chilisa et al. 2015, 
pp. 317-318; Sibanda & Ofir 2021, p. 50). Adopting ubuntu allows all to 
recognise a shared environment and shared sense of wellness and 
requires a change in our embedded attitudes towards the vulnerable 
and most marginalised, as well as nature. In applying ubuntu to evaluation 
planning, implementation and use, evaluators and commissioners are 
urged to give space to the perspectives of all stakeholders and consider 
the multiple interconnected ways of knowing and being that influence 
how an intervention is experienced and how values are determined to 
ensure that benefits accrue equitably (Billman 2019; Chilisa et al. 2015, 
pp. 318-319; Chilisa et al. 2015, pp. 318-319). 

4. Inclusivity: Drawing from the literature on participatory approaches to 
evaluation (e.g. Abbot & Guijt 1998; Mertens 1999; Post et al. 2016), 
inclusivity refers to the intentional inclusion of all identities and 
geographies that are affected (directly or indirectly), ensuring that these 
are represented in the different phases of the evaluation including 
preparation, implementation and follow-up (Robinson, Fisher & Strike 
2014, p. 5). Processes need to be established to ensure that those who 
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may have been previously marginalised from evaluation processes or 
have had less voice are given the opportunity to contribute to the 
process meaningfully. For example, community stakeholders must be 
supported to have a strong voice, not merely as sources of data or token 
representatives on ‘advisory councils’. Inclusivity also means the 
priorities, interests, voices, insights and concerns of stakeholders are 
solicited and reflected upon from the conception stage through to 
delivering the evaluation findings. 

5. Systems thinking: Development interventions are often complex, with 
multiple influences and stakeholders and complex theories of change 
(Chaplowe & Hejnowicz 2021, p. 3). It is important to understand how 
interventions are part of bigger systems, with often unintended outcomes 
in linked systems (e.g. nature) (Uitto 2021, p. 92; see Footprint Evaluations 
on Better Evaluations website*’). A systems thinking approach aims to 
map and understand the entire spectrum of relationships, interests and 
influences that have a bearing on an intervention and its effectiveness 
(Hummelbrunner 2011, p. 399). In this way, evaluation practice needs to 
incorporate analyses of interactions and inclusivity within systems. This 
can be achieved by mapping and understanding the components of 
sub-systems of interest and understanding their needed contributions 
within the overall system. 


The guideline was drafted with the understanding that commissioners of 
evaluations and evaluators have different roles and perspectives in the 
evaluation process. Therefore, the application of these principles will vary 
slightly based on one’s place (of power) and role in the evaluation. Table 11.1 
presents how the equity principles could be applied by those commissioning 
and those conducting an evaluation. 


39. www.betterevaluation.org/our-work/footprint-evaluation-project 
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TABLE 11.1: Applying equity principles as a commissioner or an evaluator. 


Evaluation commissioners 


Evaluators 


Evaluation commissioners has a responsibility to 
promote equity in the process of commissioning 
and managing evaluations by: 


Adopting a mindset of social justice and equality 
and a willingness to examine structures that 
perpetuate inequities, even where these affect 
dominant interests 


Being attentive to the power dynamics at play 

in how interventions are funded, designed and 
implemented; how evaluations are carried out 
and evaluators are perceived; and how evaluation 
results are presented and used 


Ensuring both the intervention and evaluation 
process do not reinforce inequity or produce 
unintended consequences that perpetuate 
inequality and inequity 


Ensuring the evaluation asks broader questions 
about relevance and coherence in the broader 
system, and how effectively the intervention 
supports systemic change 


Establishing multiple mechanisms for stakeholder 
engagement, ensuring that such engagements 
are inclusive, respect the dignity of all, and 

give stakeholders meaningful opportunities for 
contribution by all throughout the evaluation 
process 


Welcoming divergent views to enrich the debate 
while also facilitating a process of coming to 
agreement*? 


Establishing trust and buy-in with communities, 
particularly with those that have previously been 
consulted without seeing any changes in their lives 


Ensuring that the evaluation team is diverse 

in background and experience, and able to 
understand and empathise with the most affected 
groups in a particular evaluation 


Source: Adapted from DPME (2022). 


Evaluators have a responsibility to promote and 
uphold equity throughout the evaluation process 
by: 

Self-reflecting on one’s own biases and position of 
privilege, and transparency as to how these may 
influence the evaluation 


Adopting a mindset of social justice and a 
commitment to using evaluation to promote 
Transformative Equity 


Considering the application of evaluation 

theories or methodologies that are inclusive 

and participatory, for example, participatory 
evaluation,*° feminist evaluation“ or empowerment 
evaluation’? 


Building teams that include previously 
disadvantaged individuals, thereby allowing them 
to play a more meaningful role in the evaluation 
process 


Appreciating the African context, particularly the 
importance of relationships, material and non- 
material interactions, the interconnectedness with 
nature, and acknowledgement of multiple ways of 
knowing and being 


Creating an inclusive environment where recipients 
of the intervention can meaningfully input into the 
evaluation 


Upholding ethical evaluation practice through 
ensuring that evaluation participants have full 
understanding of their rights as an evaluation 
participant, and respecting participants’ human 
rights and dignity 


40. https://www.betterevaluation.org/en/plan/approach/participatory_evaluation 


41. https://www.betterevaluation.org/en/themes/feminist_evaluation 


42. https://www.betterevaluation.org/en/plan/approach/empowerment_evaluation 


43. An example of this is in this video of the Diagnostic Review of Violence Against Women and Children 


https:/youtu.be/JFZAnEOWARA and this 


policy brief https:/wiredspace.wits.ac.za/bitstream/id/ 


e627b871-2636-4369-a5c8-79074f22bd56/Government%20Commissioned%20Evaluation_Building%20 
a%20culture%200f%20evidence%20informed%20policy.pdf. 
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E Integrating the equity dimensions and 
principles into the evaluation process 


For evaluations to assess the extent to which an intervention can contribute 
to the transformation of society, the evaluation questions and evaluation 
design must take stock of the specific dynamics in which inequities play 
out in the given context and the differential experiences and needs of the 
various stakeholders and participants involved. Using the structure of 
the DPME guideline on developing TORs, the hackathon team worked to 
ensure that equity would be considered at every stage, from determining 
the objective and scope of the evaluation, identifying and engaging 
stakeholders and drafting evaluation questions, to using the evaluation 
findings and the development of improvement plans. This section provides 
practical guidance on how the equity dimensions and principles can be 
brought into these elements of evaluations. 


Determining the purpose and scope of the 
evaluation 


The evaluation purpose provides the overall intention of the evaluation, 
thereby providing the basis for the evaluation scope, questions and 
methodology. The Transformative Equity guideline urges commissioners 
and evaluators to consider the equity implications of the intervention being 
evaluated (the evaluand) and consider whether the evaluation purpose 
should specifically cover equity, even if the intervention does not explicitly 
have an equity objective. To do so, commissioners and evaluators can 
consider who is included and excluded, how different groups (may) 
experience an intervention based on their context, and what spatial or 
structural factors affect the efficacy of the intervention. 


Evaluation questions 


Each evaluation purpose requires a specific evaluation type and 
particular evaluation questions relevant to the phase in the intervention’s 
life cycle. In ensuring the application of an ‘equity lens’, the guideline 
provides an example of possible questions by evaluation type, with the 
relevant dimension identified in parentheses. For each evaluation type, 
examples of detailed evaluation questions are also provided. Examples 
of evaluation questions incorporating equity by evaluation type are 
shown in Table 11.2. 
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TABLE 11.2: Suggested primary and detailed evaluation questions by type, accounting for equity. 


Type Primary evaluation questions Equity-focused evaluation questions (and in 
in the purpose brackets where these relate to equity dimensions 
and the six DAC criteria, where relevant) 
Diagnostic What is the current situation What are the equity needs or problems that 
and root cause of the problem? the intervention aims to address (how)? Has 
What structural or systemic sufficient attention been paid to the way in which 
inequities contribute to that the experienced needs or problems may differ 
problem, and what are possible between population groups (who) or geographic 
interventions to address it? locations (where), and how the problem manifests 
(how)? Has the problem been understood at a 
macro level before being looked at from the more 
micro level (what)? (Relevance) 
Design Is the logic of the intervention Is the design of the intervention inclusive, 


Implementation/ 
Process 


Outcome 


Impact 


Economic 


design relevant, robust and 
likely to work and how will it 
contribute to Transformative 
Equity? 


Is the intervention being 
implemented as specified; 

are the outcomes likely to 

be achieved and why; and 

is the intervention likely to 
result in changes in the equity 
dimensions? 


Have short-term outcomes 
been achieved as a result of 
the intervention, and what 
have been Transformative 
Equity outcomes, intended or 
unintended? 


What have been the direct 

and indirect results of the 
intervention? Have there been 
impacts on systemic inequity? 
What have been the intended 
and unintended impacts of 

the intervention, and how do 
these relate to Transformative 
Equity? 

What are the costs in relation 
to social benefits? What are the 
social costs of inaction? Is the 
programme providing value for 
money? 


Source: Adapted from DPME (2022). 


addressing societal inequities and the need for 
Transformative Equity (what)? Does it consider 
how the intervention supports or detracts from 
other efforts? Was equity considered in the 
intervention design (represented in the ToC, 
indicators, log frame, etc.)? What, if any, are the 
assumptions made in the design of the programme 
(how)? (Appropriateness and coherence) 


What factors influence the way the programmes 
are implemented (how)? What power dynamics 
are at play (how)? (Effectiveness) 


How is the context or situation changing for 
participants and stakeholders over the period of 
the implementation? What influence does a shift 
in socio-political context or discourse have on the 
intervention’s likely success (when)? 


Do the emerging outcomes match the theory 

of change in terms of how the intervention was 
intended to promote equity (what)? How do these 
outcomes differ between population groups (who), 
geographic locations (where) and contexts (how)? 
What are the reasons and explanations for these 
differences? Are these outcomes likely to lead to 
systemic and sustainable change? (Effectiveness) 


To what extent do the outcomes address the 
symptoms and causes of inequality and inequity 
(what)? Are they systemic and sustainable in 

that will they make a lasting change for the 
beneficiaries of intervention? (when) (Appropriate 
for evaluation of individual interventions) (Impact 
and sustainability) 


What is the net social benefit resulting from a 
programme (what and how)? How should this be 
viewed from an equality and equity perspective? 
(Impact) 
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In addition to the evaluation purpose and evaluation questions, an 
evaluation TOR should also provide guidance regarding the expected 
evaluation design or approach and methodology to be applied. The 
Transformative Equity guideline urges that evaluation commissioners, 
technical working group (TWG) and evaluators ensure that the evaluation 
design is inclusive, considers system effects, and gives attention to the 
power dynamics between groups and possible conflicts that may emerge 
in the evaluation. The guideline offers some reflective questions to support 
this thinking: 


e How will systemic issues of empowerment and disempowerment be 
analysed? 

e How will different populations be included in the data-collection 
process? Does the evaluation include people of different ages, classes, 
races, cultures, ethnicities, families, incomes, languages, locations, 
abilities, health and sexualities? 

e Are the data-collection strategies appropriate for diverse groups and 
diverse contexts, including providing for preferred modes of 
communication? This question requires consideration of issues such as 
language, accessibility and technical literacy. Do they encourage 
participatory methods which are culturally appropriate, perhaps using 
existing community structures or groupings?*4 

e How will the power differentials between evaluators and participants be 
managed so that bias is minimised? How will respondents be affected 
by being interviewed by male or female, younger or older, black or white 
interviewers? This potential bias underscores why it is critical for 
evaluation teams to be diverse and for evaluation teams to engage in 
ongoing self-reflection of their own biases and assumptions. 


Stakeholder identification and engagement 
process 


Given that equity is a normative, value-based concept, it is important for 
evaluations to identify which groups are of concern, which groups are 
particularly marginalised or vulnerable in the applicable context, which 
stakeholder characteristics are of interest and what is meant by a fair 
distribution. Intentional and meaningful engagement of diverse 
stakeholders remains critical to the success of evaluations and the 
usefulness of the evidence produced. This emphasis on meaningful 
engagement of stakeholders is particularly relevant in the South African 
NES, which has adopted a utilisation-focused approach in which 


44. See for example https://www.participatorymethods.org/glossary/participatory-learning-and-action- 
pla. 
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evaluations are undertaken for and with the primary users in mind (Patton 
2008). This means that programme participants are key stakeholders in 
the entire evaluation process. 


The Transformative Equity guideline provides several suggestions for 
enhancing inclusive and equitable stakeholder engagement. A stakeholder 
analysis should be conducted to ensure that stakeholders and participants 
with diverse characteristics** affected by the intervention are included in 
the evaluation process. This intentional inclusion ensures that the voices 
of those affected by the intervention meaningfully participate in the 
evaluation planning, design, implementation and discussion of findings 
and recommendations. The guideline also states that representatives from 
the most affected and marginalised groups should be included on the 
Evaluation Steering Committee (ESC) or other similar oversight bodies 
and involved in such a way that they can make a meaningful contribution.*© 
The ESC also holds responsibility for ensuring equitable participatory 
practices during the evaluation process. Finally, guidance is provided to 
commissioners and evaluators regarding processes that can be used in 
different stakeholder meetings to ensure inclusion and equity, including 
ensuring that language and presentation are inclusive and accessible, rules 
are established to equalise speaking time, and multiple meetings times are 
provided to enable broader participation. 


The validation of evaluation findings presents another valuable 
opportunity to engage stakeholders in the review and interpretation of 
results and how these results can be used for improved interventions. The 
guideline states that participants of this workshop should be carefully 
selected to include groups differently affected by the intervention as well 
as stakeholders with diverse views. The process should encourage 
participation by all groups and create meaningful interaction with the 
findings and the potential to make recommendations. 


Bringing equity into the post-evaluation 
processes 
The South African NES was established purposively to maximise the 


likelihood of use (NEPF of 2011, p2 and 4). The objective of having a 
Transformative Equity criterion is that evaluations promote system-level 


45. Such as abilities, ages, classes, cultures, ethnicities, families, incomes, languages, locations, races and 
sexualities. 


46. For example, if an evaluation related to HIV and AIDS this may mean representatives from a group such 
as an association of home-based carers, or if around support to small business, representatives of informal 
traders. The process by which they are involved also matters to ensure they have an equal voice. 
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change that contributes to greater equity within society. Evaluations are 
only of value if they are used, and so the follow-up process is critical. 
Specifically, this includes communication of the evaluation results to 
stakeholders and participants and the development of the improvement 
plan. Goldman and Pabari (2020) unpack the lessons of what drives 
evidence use, drawing from five African countries. They provide a table 
indicating the types of interventions that can be taken to support evaluation 
use, found in these countries’ NES, and other interventions promoting use. 
Many of these elements are present in the South African NES. 


In keeping with the principles of inclusivity and participation, it is very 
important that results from evaluations are shared timeously with the 
public, particularly participants and those affected, and in accessible ways 
to different stakeholders (considering formats, language, media, etc.). In 
view of the power dynamics within communities, commissioners and 
evaluators must also consider the key gatekeepers who will enable or 
block communication. Community members may be able to take the 
findings from the data analysis and develop their own follow-up 
interventions, and this feedback is very useful in refining an evaluation 
report or preparing an improvement plan. Critically, it is important that 
those affected by the intervention have an opportunity to contribute to 
developing the improvement plan. However, historically, efforts to 
communicate evaluation results are underplayed in South Africa, with the 
main focus being on formal reporting upwards and little budget allocated.*” 
This needs to change. 


E The pilot process: Mainstreaming the 
criterion into practice 


The intended outcome of the hackathon process was that the two criteria*® 
and guidelines be applied to all future evaluations, thus transforming the 
evaluation system to be more intentional regarding issues of equity and 
CEH. In order to test this, the decision was taken to pilot the practical 
application of the guideline in real evaluations. Targeted recruitment was 
used, seeking up to six evaluations that had TORs drafted andimplementation 
scheduled for late 2022 into early 2023, asking government and non- 
governmental organisation (NGO) partners. In the end, only three 
evaluations were found at this stage: (1) the Unemployment Insurance Fund 
(UIF) DPME impact evaluation on the COVID-19 TERS grant; (2) the DSD 


47. The evaluation of the national evaluation system found that less than 1% of budgets are spent on 
communication (Goldman et al. 2019). 


48. Including the climate and ecosystems health criterion. 
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design and implementation evaluation of the Household Food and Nutrition 
Security Programme; and (3) the Department of Science and Innovation 
(DSI) implementation evaluation of the Water Research, Development and 
Innovation Roadmap. Teams from all three commissioning departments 
comprised a mix of M&E and programme staff, participated in a training 
conducted in July 2022. During the training, participating groups engaged 
with each criterion and considered how evaluation questions and scope 
could be adjusted to include the criteria. The three evaluations proved to 
be at different stages in their process, making the training and piloting 
more complex and generating interesting lessons. 


For the Household Food and Nutrition Security Programme, it was 
discovered during the training that the TORs had already been approved, 
precluding fundamental shifts and changes to the TORs. The team 
recognised the relevance of the two criteria to the evaluation of the 
programme and agreed to incorporate the criteria during the inception 
phase when evaluation questions and methodology can be adjusted. In 
early December 2022, SAMEA board members engaged with the contracted 
evaluators to sensitise them on integrating the criteria into the evaluation 
questions and process. Although the evaluation process has experienced 
some procedural delays, the DSD continues to engage with SAMEA on the 
criteria and has indicated keen interest in institutionalising just transition 
into their M&E processes using the two guidelines. 


In the DSI case, the TORs were adapted following the training to include 
questions to address both equity and CEH before being presented to the 
Bid Specification Committee (BSC) for approval. The evaluation manager 
noted that the guidelines assisted in developing appropriate questions, 
such as ‘Has the intervention been responsive to vulnerable groups and 
institutions, namely: females, disabled, youth, historically disadvantaged 
institutions and disadvantaged communities? If not, how can this be 
achieved?’. However, they also noted that queries were raised by the BSC, 
as they recognised that current project had not collected data relevant to 
answer some of the additional questions. This gap speaks to the need to 
sensitise programme staff as well as M&E staff on these issues for integration 
in the design phase. In addition to adaptations to the TORs, DSI has made 
considerable progress in introducing and integrating the criteria and 
guidelines into their departmental processes and aim to include them in 
the departmental M&E framework. 


Finally, during the training, the participants for the COVID-19 TERs 
impact evaluation were challenged to think of how to apply some of the 
dimensions of the criteria to their evaluation design. Specifically, the team 
struggled to see how the criterion on CEH was relevant to unemployment 
schemes. For reasons unrelated to this pilot process, this evaluation has 
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not progressed beyond the draft TORs stage, and therefore, reflections on 
the learning from this example are limited. 


The piloting process is generating lessons for SAMEA, DPME, DSD and 
others engaged in this development and testing process. Initial lessons 
learned include: 


e The relevance of an organisational champion for the issue - in this case 
DSD, who have a relevant mandate to leave no one behind - and 
evaluation capacity, but also a need to ensure this is not seen as a 
sectoral issue. 

e The importance of a champion in the commissioning organisation or 
department of the evaluation, ideally from both M&E and programme 
staff, who will internalise the topic and drive the inclusion of equity at all 
stages. 

e The importance of building proper reflective processes into the TOR 
development and evaluation implementation stages. People quickly 
make judgements on the relevance of these topics without thinking on 
them systematically. Shallow reflection will not address the transformative 
changes needed to address the magnitude of the equity (and climate 
and ecosystems) crises. With South Africa’s history, it proved easier for 
departments to take on equity issues, which are keenly felt by most 
government staff, rather than climate and ecosystems issues. 

e The importance of balancing the number of evaluation questions to 
adequately address the overall purpose of the planned evaluation while 
also accounting for the need to make the equity intention explicit. 

e The need for tools, checklists and rubrics to support the application of 
the criteria and guidelines. 

e Incorporating gender-responsive and equity standards within the 
evaluation quality assurance processes is one way of ensuring that 
government evaluators contribute towards addressing social justice 
through evaluations. Yet, commissioners, programme staff, bid 
committees and others need more sensitisation on how these criteria 
and guidelines are supporting tools in this process. 


In addition to these lessons, the authors anticipate the need for much more 
advocacy and capacity-building on the criteria and guidelines to actuate 
the theory of change. Through the process of developing the criteria and 
guidelines, a community of practice has emerged to facilitate greater 
learning and application within the M&E sector, and SAMEA has decided to 
continue with this theme over the coming years. In the coming years, the 
authors anticipate this area of work to further develop as issues of 
Transformative Equity and CEH become more embedded in the NES. 
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E Conclusion 


The criterion and guideline described in this chapter have been developed 
from the perspective that evaluation can and should play a critical role in 
societal transformation, particularly in addressing the systemic inequities 
that persist in South Africa. The current DAC criteria that are commonly 
used in evaluations across the world do not adequately cover issues of 
equity. Without an explicit criterion on equity, evaluators and commissioners 
of evaluation may overlook key questions and steps necessary to ensure 
that issues of equity are adequately and purposefully included during the 
evaluation process. 


The co-development of the two evaluation criteria and guidelines 
addressing Transformative Equity and CEH and the commitment of DPME 
to issue the guidelines has resulted in a major step to influence evaluation 
to better support the desired develooment outcomes and impact on 
society as envisioned in the NDP. It is still early to determine whether the 
hypothesis has worked and whether the application of the guidelines will 
be mainstreamed and have the transformative effect intended. The process 
to date has clearly raised the profile of Transformative Equity and fostered 
the commitment of DPME, DSI and DSD to institutionalise the work. 


Early learnings are showing the need for greater advocacy on the value 
of a dedicated criterion on equity and more capacity-building for 
commissioners and practitioners for how to use the criterion in practice. 
More intensive support is needed over the next year to test the criteria and 
guidelines, to support the pilots effectively, document the lessons and 
establish how to build capacity to apply these widely in South Africa. The 
SAMEA will be pursuing this work over the next three years, engaging the 
emerging community of practice focused on evaluation for just transition 
to develop more knowledge, skills and practical tools within the evaluation 
sector and government departments to learn together how M&E can best 
contribute towards a just transition to a more equal and sustainable world. 
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E Introduction 


Evaluation is regarded as a powerful tool that can be used to address 
several socio-economic issues and improve the effectiveness of 
development programmes. However, several scholars have highlighted the 
limits of evaluation as a tool. Challenges include evaluations being 
accountability-focused, the disjuncture between evaluation theory and 
practice and the project mentality that is widely adopted in evaluations. 
These factors have limited the role of evaluation in transforming society 
and improving the development outcomes of interventions. Several 
evaluation studies have shown that by being accountability-focused, the 
commissioners of evaluations and evaluators miss the key component of 
learning and applying evaluative thinking to the different interventions that 
are being implemented. Furthermore, evaluations have failed to capture 
the several components that shape the life experiences of the programme 
beneficiaries. For evaluation to be an effective tool, it needs to be critical, 
ask difficult questions and trigger internal and external reviews of 
approaches and design of interventions. This book has shown that 
evaluators need to be conscious of the prevailing social reality and 
understand how various issues interlink to produce that social reality. There 
is an inherent demand for programme designers, implementers and 
evaluators to focus on the root causes of the several challenges that are 
experienced in the Global South. What is emerging is that an equitable and 
just society cannot be achieved through simplistic approaches to evaluation 
but through complex and in-depth approaches that unravel the underlying 
mechanisms that drive unequal societies. 


This book highlights the various approaches that evaluators can use to 
evaluate interventions that seek to address inequality in the Global South. 
Evaluators are urged to adopt multipronged approaches to understand the 
underlying causes of inequalities and pass a judgement on the effect of the 
interventions seeking to address that inequality. This book made a clarion 
call to evaluators to go beyond answering the questions: ‘Are we doing the 
right thing?’ and ‘What makes this the right thing to do?’ The call is for 
evaluators to go beyond the narrow focus on projects and other interventions 
and to have a broad look at the systemic drivers of inequality and address 
them. This approach will place the intervention in its context and foster a 
broad understanding of its contribution to addressing the systemic 
injustices in the communities. 


E Evaluators’ conceptualisation of inequality 


This book alludes to the fact that although equality and equity are generally 
conflated and treated as synonyms, they are, in fact, different concepts. 
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While equality involves the provision of the exact same resources, benefits 
and opportunities to all people, groups and areas in a country, regardless of 
identity, abilities, position, privilege or need, equity refers to the provision of 
the appropriate levels of support to individuals, groups or areas, according 
to their level of need in pursuit of fairness and justice within social and 
economic systems that resulted from the historical structural inequalities 
mentioned earlier. The experiences of people in Africa of inequality are better 
articulated by the concept of ‘intersectionality’. Intersectionality refers to the 
ways in which different forms or systems of inequality based on race, class, 
gender and physical and mental abilities, among other issues, often overlap 
and thus compound the negative effects experienced by people with these 
various identities and backgrounds. 


In this book, there is a common thread that shows that the root of the 
inequalities that are experienced in the Global South are rooted in 
colonialism and its related system of oppression, slavery, racism, patriarchy 
(rooted in sexism), etc. (Obeng-Odoom 2015). These historical structural 
inequalities, which were intentional forms of discrimination and prejudice, 
bred systemic inequalities that are perpetuated and normalised through 
the institutions, regulations and practices currently in place in society and 
which continue to inform experiences of marginalisation for certain worst- 
off groups of people. The book emphasised that the Global South has not 
yet recovered from the impact of colonisation. Inequities affect most of the 
population in the Global South to the extent that interventions focus on 
those who are regarded as worse off. The worst-off groups are consistently 
identified as females, those living with physically and mentally disabilities, 
youth, children and the elderly (Bamberger & Segone 2011, p. 12), as well as 
those on the extreme end of deprivation or poverty. Most of the current 
development interventions are focused on these worst-off groups, leaving 
others to fend for themselves. 


The conceptualisation of inequality in evaluation is influenced and 
rooted in the ideological and epistemic hegemony of the Global North. The 
conceptualisation is driven by the power of development aid that is 
controlled by the Global North. The concept of development aid has its 
roots in the establishment of the Bretton Woods system in the 1940s, and 
the introduction of development aid in Africa coincides with the end of 
colonial rule and the beginning of independent governance of many African 
countries in the early 1960s by former colonial governments and 
international multilateral institutions such as the World Bank and 
International Monetary Fund (IMF), and other bilateral and multilateral 
donor organisations (Harriss 2013). This book illustrates the continued 
economic and political hegemony of these Global North powers, which 
have led to the current conceptualisation of what development should look 
like in Africa and how it is predominantly measured. Of concern is the 
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limited attention paid to the views of the communities in the Global South 
when it comes to the conceptualisation and design of interventions that 
seek to address their challenges. Evaluations are largely pitched at 
addressing the information needs of the funders from the Global North and 
ignore the local needs. 


For development to be equitable, it needs to assume an equity lens, not 
the narrow neoliberal perspective of development that is aligned with 
economic growth centred on market-oriented reform policies such as 
‘eliminating price controls, deregulating capital markets, lowering trade 
barriers’, especially through privatisation and austerity, state influence in 
the economy and reducing spending on priority sectors and issues. The 
notion that economic growth is a panacea to addressing inequality has 
been proven to be less effective in reducing inequality. This book also calls 
for a need to realise that the contextual nuances of the Global South 
communities are different from those of the West; hence, pitching 
development interventions that are based on Western notions of 
development and values might result in more harm in the Global South. 


The book also posits that the perpetuation of inequality is linked to 
Africa’s dependence on foreign aid, which entrenches neoliberal 
development ideals. The role of colonial history and how it impoverished 
Global South communities is not considered, and the implication is that 
the conceptualisation of ‘development’ neglects the very reason why 
development is needed. Thus, inequality in this book is understood as any 
imbalance of power, including development decision-making power and 
prioritisation of development interventions. The power asymmetries are 
witnessed through the influence of global powers and donor institutions on 
sectoral agendas and priorities in Africa. The most concerning is that there 
appears to be a lack of a contextual, systems thinking approach and a 
holistic approach to addressing various inequality issues in the Global 
South. Instead, the interventions focus on addressing the symptoms of 
inequality rather than its root causes, thereby entrenching the systemic 
structural inequalities and the privilege of the Global North. 


Further, social inequalities are viewed through unequal power dynamics 
between males and females, generally perceived as a key driver of gender- 
based violence (GBV); however, a more nuanced conceptualisation of 
gender inequality also includes issues of discrimination as critical to 
understanding the experiences of females. Discrimination denotes the 
intersection between power and prejudice, such as racial and disability 
discrimination. Inequalities in mental health illustrate the social and 
structural disparities between population groups that are avoidable, 
unfairly generated and maintained by unjust social arrangements that 
offend common notions of fairness and keep a particular group of society 
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deprived of access to mental health services. These examples show that 
inequality cannot be addressed through efforts to achieve equality but 
rather would most effectively be resolved through efforts to achieve equity. 
This book has shown that these contextual and social intricacies are not 
fully considered in the design, conceptualisation and evaluation phases 
of interventions. 


There is a plethora of evidence that shows that the Western notion of 
development and the associated actions to address development 
inequalities have not resulted in progress or change in African countries, 
including in terms of economic growth (Harris 2017; Njoroge 2018). It is 
argued, therefore, that to be successful, development interventions and 
their evaluations should assume an equity lens. The pursuit of equity is 
important because it demonstrates the understanding that inequality is 
context-specific and varies across different individuals, spaces and time. 
Equity thus encompasses a wide variety of concepts and principles 
believed to be key to achieving social justice and fairness in Africa, 
including respect for human rights and dignity, intersectionality, 
inclusivity and diversity, recognition of power and giving voice to 
marginalised groups, ubuntu and systems thinking. These notions and 
principles should apply to both the beneficiaries of development 
programmes and also to African evaluators, to truly decolonise evaluation 
practice on the continent and make it contextually relevant and supportive 
of equitable development outcomes. 


E Proposed approaches to address inequality 


This book has shown that programme evaluation can be used as an 
effective tool to address social change and transformation in the 
development space. However, this can only be achieved through 
concerted efforts to adopt approaches that are relevant and lead to a 
better understanding of the problem and its context. The argument 
posited in this book is that development should be viewed as an equity 
project, and associated evaluations should adopt an equity lens. However, 
to achieve this, there is a need for the decolonisation of African evaluation 
practice, and the curriculum should be linked to this effort, as this 
approach would be more context-responsive. This book also argues that 
the inclusion of African indigenous knowledge, worldviews, experiences, 
values and aspirations into public policy and development planning, 
programme implementation and evaluation can go a long way in creating 
a just and equitable society. The Made in Africa Evaluation (MAE) 
framework provides an array of approaches that can allow evaluators to 
grapple with key challenges relating to hierarchy, power asymmetries and 
representation in international development. In addition, giving voice to 
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African beneficiaries and their experiences is seen as a more plausible 
approach to improving development interventions. 


Furthermore, this book highlights the importance of adopting the 
‘Intersectional Framework’ as a tool for unpacking inequality as it would 
provide a more comprehensive explanation of the inequalities faced by 
African people (race, gender, identities and mental and physical abilities, 
etc.) and enable evaluators to focus on the broader root causes of the 
problem. The ‘Intersectional Framework’ in evaluation contributes to a 
better understanding of chronic development challenges such as climate 
injustice, poverty and income inequality. It enables programme designers 
to come up with holistic interventions that effectively address a wide 
array of injustices that limit communities’ and individuals’ resilience, 
prosperity and well-being. The adoption of the intersectional lens 
necessitates evaluators not to limit themselves to the subject at hand but 
to focus on the broader root causes of the problem. This will enable 
programme designers to come up with holistic interventions that focus 
on a wide array of injustices. By avoiding examining the root causes of 
development challenges, evaluation will continue to scratch the surface 
and view problems using a narrow focus. Communities have ingrained 
injustices that cannot be solved by glossing over contextual factors that 
perpetuate them. 


In addition, this book also advocates for the incorporation of an equity 
criterion into evaluation practice. The decision to create this criterion is 
based on the lack of satisfaction with the argument of the Organization for 
Economic Co-operation and Development - Development Assistance 
Committee (OECD-DAC) explanatory notes that though equity is not 
explicitly addressed by these widely utilised evaluation guidelines, they are 
implicitly addressed by a few of the criteria. Before evaluators and 
commissioners of evaluation can think about designs and methods, they 
first need to decide on the right evaluation questions to ask, and these can 
be informed by this criterion and its guidelines. Guided by an equity- 
focused criterion and equity principles to evaluation, evaluators are then 
encouraged to reject the approach of selecting evaluation designs and 
methods based on Western epistemic hierarchies, where randomised 
control trials (RCTs) are regarded as the ‘gold standard’ and are at the top 
of the methodological hierarchy. The RCTs are riddled with flaws and 
weaknesses that are often not acknowledged, and several chapters across 
this book call this into question and propose other approaches that are 
more suitable and contextually relevant. 


Another good example of an approach to equity-focused evaluation 
that can adequately capture issues of inequality is ‘Critical Discourse 
Analysis’ (CDA). The CDA looks at the social aspect of utterances (by the 
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beneficiaries, funders and the implementers during evaluation) in terms of 
the socio-economic and political situation on the ground. The argument 
put forward in this book is that this complementary framework brings the 
much-needed dynamism to decode the hidden language that forms the 
basis of prolonging inequality. This is achieved by providing a critical view 
that identifies power relations, struggles of power and the systems that 
maintain power relations. 


This book also highlights that certain inequalities, such as those 
related to gender, may be more appropriately studied using analytical 
frameworks such as the ‘What is the Problem Represented to be?’ (WPR) 
framework. This is a feminist, post-constructivist policy analysis method 
that may be used to interrogate the conceptualisation of gender 
inequality in GBV interventions and evaluations. The WPR framework 
suggests that problems are not self-evident and do not objectively exist. 
Rather, problems are framed in a specific location and in the context of 
existing power relations. The WPR framework calls for attention to be 
paid to the importance of power and the effects of power that are at 
the centre of how problems are defined and acted on. The WPR analytic 
framework not only focuses on how problems are spoken of (discourse 
analysis) but is primarily concerned with ‘the outcomes and material 
implications that arise when phenomena are constituted in particular 
ways as particular kinds of problems’. 


Finally, this book also proposes the use of agenda setting (refers to how 
a particular issue gains the attention of policy-makers among other issues 
competing for priority) as an approach to get the issues of inequality on 
the development agenda. The agenda setting enhances our understanding 
of how problems are identified and defined. When agenda setting is 
coupled with social theory framing, it helps to deconstruct the process by 
which actors construct and represent meaning, thereby enhancing our 
understanding of social reality. 


E Reflection on evaluation approaches 


One of the long-standing and significant challenges affecting evaluation 
contribution to addressing inequality in Africa is the long-standing reliance 
on Western/Eurocentric evaluation curriculum. Inevitably, this over-reliance 
has produced evaluators who mimic Western ideals in evaluation and 
programme design. Thus, there is a concerted need to address these 
challenges at this level by developing and advancing an African evaluation 
curriculum that responds to the need to confront issues of inequality in 
evaluation practice. Pertinent to this is the lack of adequate tools, 
methodologies and criteria for measuring and assessing the extent to 
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which development interventions achieve equity and social justice in Africa. 
One of the critical methodological challenges in addressing inequality and 
inequity in evaluation is the lack of explicit inclusion of equity in the most 
well-known and widely used OECD-DAC criteria. As evaluation criteria 
guide the development of evaluation questions, they are considered 
important guideposts when conducting evaluations. Thus, if evaluators 
and commissioners of evaluation do not explicitly consider equity, they 
are most likely to overlook these issues, resulting in inadequate and 
purposeless evaluations. 


Similarly, evaluators need to pay more attention to the methods used to 
evaluate achievement of develooment outcomes in an equitable and fair 
manner, taking into consideration the nuanced and culturally appropriate 
designs to capture the development context. Ultimately, approaches, 
designs and methodologies should contribute to shifting current thinking 
about the interventions and options used to address inequities. A key 
methodological challenge prevalent in African evaluative evidence points 
to the over-reliance on quantitative and experimental methodologies to 
evaluate and measure inequality. In particular, the use of RCTs adopted as 
the ‘Gold standard’ in multiple sectors to evaluate the impact of development 
interventions. They are regarded as objective and value-free. Randomised 
Control Trials are framed as value-laden; however, the measurement tools 
utilised in the assessment of outcomes and impact are influenced by 
subjective factors, including which elements are investigated and evaluation 
questions are asked. More worrying is the fact that these evaluation designs 
tend to examine single issues in complicated contexts and often lack the 
capacity to have a holistic understanding of the problem and how to solve 
it. There are valid concerns about the continued use of RCTs, which ‘look 
for keys under the streetlight’, meaning they involve looking for a solution 
to a problem only where one is able to see, regardless of the likelihood that 
it is there. 


We posit that methods and approaches utilised to evaluate interventions 
are not just technical tools but rather reflect how we see the world or at 
least interpret what we observe, what we perceive as credible data and 
how that data are made sense of. More often, the decision to use certain 
evaluation methods is often determined by evidence hierarchies. 
Randomised Control Trials are at the top of the methodological hierarchy 
and, therefore, are considered the go-to method for evaluators, and the 
evidence generated from their use is perceived as the most credible and 
appropriate for policy-making (Parkhurst 2017). Evaluation methodologies 
that utilise this methodology fail to consider the effect of compounding 
and extraneous variables on the outcomes of the interventions being 
evaluated. Evaluators must therefore question their assumptions and 
presumptions in designing evaluations and analysis. It is not just those who 
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use qualitative designs that should engage in self-criticism and reflexivity, 
but also those who use quantitative designs. In addition, interrogating the 
limitations of widely used approaches, such as theory-based approaches, 
that may reinforce inequity and injustice goals promoted by donors and 
more powerful stakeholders is important. Critical analysis of the root causes 
of the problem and boundary limitations on the problem itself, as well as 
stakeholders’ involvement in construction and validation, remains a key 
critique of such approaches to ensuring the inclusion of equity and justice 
considerations. 


It is imperative that the process of generating evidence is scrutinised, 
but it is equally important to evaluate the quality of the evidence produced. 
For example, acknowledging that the randomisation design used in RCTs is 
problematic and is not enough, particularly when evaluating public sector 
programmes because government efforts are directed towards addressing 
colonial and post-colonial injustices by improving conditions for the poorest 
and underserved communities. Questioning the evidence produced 
through randomisation that excludes the same vulnerable groups of people 
from which policy-makers need information to make informed decisions is 
also important. This should be done particularly in cases where it has been 
shown in the literature that the under-prioritisation of vulnerable groups in 
African governments is linked to a lack of data revealing the severity and 
burden of diseases. 


Unlocking the value of evaluation in addressing 
inequality 


Made in Africa Evaluation principles that promote equity should inform and 
guide evaluation theory and practice in Africa. This will ensure that a 
plurality of context-shared values whose values do not supersede other 
values are included, among other pertinent issues impeding evaluations’ 
contribution to addressing inequality in Africa. Development, in its true 
sense, is based on the balancing act of the many competing forms of 
inequality. Applying a multiple and intersecting lens helps to see how a 
problem that manifests in a certain community can result from factors 
outside of the community. The importance of merging the evaluator and 
the beneficiary context cannot be understated. The current practice is that 
an evaluation is never shaped by the beneficiary context but by the context 
of the commissioner and the evaluator. Hence, the need for an intersectional 
lens during evaluation to promote justice. Understandably, addressing one 
form of equality might have an adverse impact on other forms of inequality 
experienced by the same group of people, community or individual. Even 
more critical is the need for evaluators to have the cultural competence to 
manage tensions within this context. 
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In addition, the focus on programmes as the unit of analysis in an evaluation 
can also lead toa narrow definition of the issue and impede the programmes’ 
ability to address inequity. In order to truly resolve systemic issues, a ‘whole 
of society’ approach is required, including promoting the mainstreaming of 
intersecting issues at multiple levels, impeding the realisation of a just and 
equitable society into all policies and programmes. Thus, application of the 
principles of the WPR, CDA, Agenda and Framing frameworks extending 
beyond obvious and simplistic theories and narratives instead prompting 
for evaluation to question how problems are represented, which voices are 
dominant and which perspectives are undermined, can go a long way in 
ensuring that evaluations and evaluative enquiry identify the systems that 
are driving social problems and therefore better inform the design of 
interventions that can transform such systems. Thus, evaluations must 
begin integrating concerns of power, ideology and politics in the evaluation 
of development interventions. Formulating questions and approaches 
that lend themselves to inquiring about the types of outcomes that are 
desirable, as well as whether and how they are privileged over others, is 
required. The Transformative Equity criterion and accompanying guideline 
and the pilot projects used to test the integration of the criterion into terms 
of reference and evaluation implementation processes of various South 
African government interventions ought to include such tools and 
processes. If indeed, these guidelines and criteria are to contribute to MAE. 
Nonetheless, the guideline provides a useful framework guiding the 
operational use of five dimensions of the Transformative Equity criterion, 
which can be viewed as critical to understanding the socio-historical 
context of South Africa and factors that reproduce inequity and promote 
transformation. 


Finally, there needs to be greater recognition that the concerns of 
subjectivity contribute to the marginalisation of other forms of knowledge 
and, in turn, promote the blind use of as well as expanding Global North- 
South power and knowledge creation asymmetries. As an example, whilst 
it is important to advance MAE, it is critical to address challenges associated 
with the overwhelming influence of quantitative evaluation designs like 
RCTs. The first step requires evaluators’ awareness and acknowledgement 
of the flawed nature of RCTs in order to raise awareness of the need for 
equal treatment of this evaluation method and the dismantling of the 
epistemic hierarchy. Following this, the lessons from the weaknesses and 
failures of RCTs ought to be reflected in evaluation literature, which is 
currently limited by the ‘intellectual dependency’ of African scholars and 
policy-makers on trends set by the Global North. Indeed, it is well past time 
for more epistemological critique to be generated and disseminated from 
the Global South on matters that affect its people so profoundly. 
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E Conclusion 


This book has shown that for inequality to be addressed there is a need 
for a systematic approach. However, the current configuration of the 
stakeholders involved in international development does not allow a 
systematic approach to address the myriad of inequalities. There is a need 
for further research that can come up with approaches that enable the 
integration of interventions and synergies between the various players in 
the international development space. Secondly, more research should be 
conducted on tools and approaches that can improve the low quality of 
evaluative studies conducted in Africa, limiting studies inclusion in critical 
synthesis of development interventions. Thus, evaluators need to be 
empowered with critical analysis tools to ensure that evaluations contribute 
to addressing equity as part of the MAE Toolbox for a range of development 
interventions. Thirdly, there is a need for further contextualisation of the 
evaluation approaches and the popular OECD-DAC criteria to the Global 
South context. As highlighted earlier, the current practice is to use the 
criteria as it is despite the identified weaknesses. 
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This scholarly book is a remarkable exploration of the profound impact of COVID-19 on learners’ holistic 
development and the crucial support they require. This publication offers a profoundly insightful and 
culturally sensitive analysis incorporating diverse theories, including Afrocentric perspectives and an 
ubuntu lens. 

The book stands out for its pioneering focus on the multifaceted effects of COVID-19 on learners’ 
development and the indispensable support systems that must be in place. The inclusion of lesser- 
known theories, such as Nsamenang's social ontogenesis theory, elevates the discourse, making it an 
exceptionally compelling and intellectually stimulating read. Moreover, the meticulously structured 
framework and thoughtfully curated chapters provide a panoramic view of the subject matter, leaving no 
aspect unexplored. 

Through scientifically substantiated research and meticulous methodological explanations, this 
book delivers invaluable insights into the impact of COVID-19 on learners’ development. It serves as 
an indispensable academic resource for researchers and scholars, empowering them with essential 
knowledge to effectively navigate the challenges faced by learners during and beyond the pandemic. 

The editors deserve to be commended for their judicious timing in publishing this important work 
and for assembling a team of interdisciplinary researchers and authors of the highest calibre. Their 
deliberate emphasis on an African-centred approach fills a significant void in the field and highlights 
the importance of adopting an interdisciplinary lens. The exceptional standard of scholarship exhibited 
in each chapter's research projects, coupled with the comprehensive introduction and concluding 
summary chapter, further solidifies the book's impact. 

In conclusion, this publication makes a resounding contribution to our understanding of the 
profound impact of COVID-19 on learners’ development and the indispensable support they require. 
This book is an indispensable resource for scholars in education, psychology, and related fields due 
to its integration of diverse theories and meticulously constructed framework. Through the editors’ 
unwavering dedication and the book's comprehensive nature, this publication lays the foundation 
for further ground-breaking research and transformative interventions, fostering holistic learner 
development in the face of future challenges. 
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The publication of this book comes at a good time, given the endemic power imbalance between 
countries in the Global South and the Global North. The post-COVID-19 world demonstrates the 
perpetual unequal distribution of power and resources between developed and developing countries 
in the Global North and Global South. Millions of people suffer from this asymmetry, which contributes 
to growing inequality. Due to its inherent value judgement in the evaluation process and its potential 
influence on policy decisions, evaluation should speak truth to power rather than remain complicit. 

This book on equity in evaluation is a step towards distilling and synthesising the current status and 
direction of evaluation in African contexts in the 21st century. This book will most certainly be a strong 
addition to the theoretical evaluation scholarship toolbox as it will further illuminate the current status 
of evaluation in African contexts. Global South evaluation voices in this publication contribute to a larger 
body of knowledge by invoking African epistemology, ontology, and axiology, which could engender 
transformative evaluation practices. 
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